Lets step back. Since you are adding conditional #ifdefs to dense.c - its not appropriate to add #requirespackage to the corresponding makefile. [as dense.c should be compiled with or without magma]
Secondly - you should use PETSC_HAVE_MAGMA - not PETSC_HAVE_LIBMAGMA If you still have errors - send the logs again. Also send the following: dense.c preprocessed dense.c. i.e: cd src/mat/impl/dense/seq make dense.o CFLAGS=-E [the preprocessed output might be in dense.o] Satish On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > I even tried this with a fresh git pull from Hg repository. I added magma.py > for configuring with magma. Then I added only the following in dense.c > > #if defined(PETSC_HAVE_LIBMAGMA) > #if !defined(HAVE_CUBLAS) > #define HAVE_CUBLAS > #endif > #include "magma.h" > #endif > > Then I followed the procedure I mentioned below, and I can reproduce the > problem. Now I'm certain that removing #requirespackage is causing it. > > 1) remove #requirespackage from src/mat/impl/dense/seq/makefile > 2) remove build directory > 3) remove externalpackages directory > 4) configure && make > I get these errors. > 5) add #requirespackage to src/mat/impl/dense/seq/makefile > 6) remove build directory > 7) remove externalpackages directory > 8) configure && make > No errors. > > Harshad > > > > >>>>>>>> > /usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 > -fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include > -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include > -I/home/harshad/Downloads/magma-1.4.0/include > -I/home/harshad/cuda-5.5/include > -I/home/harshad/workspace/petsc-hg/include/mpiuni -o > CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c > In file included from > /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0, > from > /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12, > from /home/harshad/Downloads/magma-1.4.0/include/magma.h:15, > from > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10: > /home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: > #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. This > typically happens in Makefile.internal." > <<<<<<< > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > Attached make.log > > > > ----- Original Message ----- > > From: "Harshad Sahasrabudhe" <[email protected]> > > To: "petsc-dev" <[email protected]> > > Cc: "Barry Smith" <[email protected]> > > Sent: Wednesday, August 28, 2013 4:33:23 PM > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > I'm getting errors when compiling with MAGMA and without the > > #requirespackage tag in the makefile. It builds correctly if I have > > #requirespackage in the makefile. There are lots of these errors with > > different type names: > > > > magma-1.4.0/include/magma_s.h:607:38: error: unknown type name > > ‘magma_queue_t’ > > > > Harshad > > > > ----- Original Message ----- > > From: "Satish Balay" <[email protected]> > > To: "Harshad Sahasrabudhe" <[email protected]> > > Cc: "Barry Smith" <[email protected]>, "For users of the development > > version of PETSc" <[email protected]> > > Sent: Wednesday, August 28, 2013 4:26:20 PM > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > If you have conditional code in dense.c - then you don't need > > "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile > > > > Are you getting errors when compiling with Magma or without magma? > > What errors? > > > > Satish > > > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > Hi Barry, > > > > > > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA > > > code is protected by #if defined statements so it is only compiled when > > > we have MAGMA. > > > > > > When we have MAGMA, I get errors while compiling, if we don't have > > > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously > > > don't require MAGMA to compile dense.c. Is there any other way to tell > > > make that MAGMA lib has to be used for compiling dense.c when it is > > > present? > > > > > > I have attached the makefile inside src/mat/impls/dense/seq folder. > > > > > > Thanks, > > > Harshad > > > > > > ----- Original Message ----- > > > From: "Barry Smith" <[email protected]> > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > Cc: "For users of the development version of PETSc" > > > <[email protected]> > > > Sent: Tuesday, August 27, 2013 11:20:44 PM > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > Does the LU factorization use magma on the gpu now work? You told us that > > > for factorization magma allocated the memory on the GPU and copied the > > > data over everything over automatically. If the factorization and solve > > > stuff all works then we can talk about the MatMult() and other stuff but > > > otherwise let's get the factorization all working before moving on to the > > > "harder" stuff. > > > > > > barry > > > > > > > > > > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <[email protected]> > > > wrote: > > > > > > > Hi Barry, > > > > > > > > The functions MatCreate_SeqDense and > > > > MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation on > > > > GPU. I can see 2 ways to implement this: > > > > > > > > 1) Modify these functions so that they allocate memory on host or > > > > device when the solver is magma. I don't know how to do this. > > > > > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma > > > > matrix class which is very similar to the dense matrix class PETSc > > > > uses, with some additional flags. Modify the MatMatMult, MatGetFactor, > > > > MatSolve functions so that they use magma functions in the proper way. > > > > I can do this. > > > > > > > > Can you please advise on how I should progress? > > > > > > > > Thanks, > > > > Harshad > > > > > > > > ----- Original Message ----- > > > > From: "Barry Smith" <[email protected]> > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > Cc: "For users of the development version of PETSc" > > > > <[email protected]> > > > > Sent: Tuesday, August 27, 2013 8:43:57 AM > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > > > > Both _p_Mat and _p_Vec have void pointers that can be used for the > > > > gpu memory as well as flags that could potentially be used. See below > > > > > > > > struct _p_Mat { > > > > PETSCHEADER(struct _MatOps); > > > > …. > > > > PetscBool nooffprocentries,nooffproczerorows; > > > > #if defined(PETSC_HAVE_CUSP) > > > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the > > > > matrix on the gpu*/ > > > > #endif > > > > #if defined(PETSC_HAVE_VIENNACL) > > > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the > > > > matrix on the gpu*/ > > > > #endif > > > > void *spptr; /* pointer for special library > > > > like SuperLU */ > > > > > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe > > > > <[email protected]> wrote: > > > > > > > >> Hi Barry, > > > >> > > > >> Thanks, MatGetFactor() now much more clear. > > > >> > > > >> However, I have run into trouble with the device memory allocation for > > > >> Mat_SeqDense. How do I figure out whether the allocated memory is on > > > >> host or device in MatSeqDenseSetPreallocation_SeqDense() ? > > > >> > > > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I > > > >> need the memory allocated on the device. As the preallocation is done > > > >> in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to > > > >> write a similar function. But checking for if(!data) is not enough as > > > >> I need to make sure that the allocated memory is on the device. > > > >> > > > >> Is it a good idea to make another Mat class for magma which has 2 > > > >> pointers, one for the host memory and one for device memory (only one > > > >> of which should point to data at any given time) ? > > > >> > > > >> Harshad > > > >> > > > >> ----- Original Message ----- > > > >> From: "Barry Smith" <[email protected]> > > > >> To: "Harshad Sahasrabudhe" <[email protected]> > > > >> Cc: "For users of the development version of PETSc" > > > >> <[email protected]> > > > >> Sent: Monday, August 26, 2013 6:19:21 PM > > > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > >> > > > >> > > > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe > > > >> <[email protected]> wrote: > > > >> > > > >>> Hi Barry, > > > >>> > > > >>> Here is my analysis, please correct if I'm wrong: > > > >>> > > > >>> For using the MatGetFactor() mechanism, I need to define > > > >>> MATSOLVERMAGMA in petscmat.h. I also need to define the functions > > > >>> MatFactorGetSolverPackage_SeqDense_magma(), > > > >>> MatGetFactor_SeqDense_magma() in dense.c > > > >>> > > > >>> I'm not creating a new matrix class, so I think I don't need to add > > > >>> anything to MatRegisterAll() > > > >>> > > > >>> I don't understand the following: > > > >>> How does PETSc figure out that there is an additional solver present? > > > >>> Do I need to define any other variables or add anything for this? > > > >> > > > >> In MatCreate_SeqDense() note the line > > > >> ierr = > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > > > >> > > > >> you will add an additional line (protected by ifdef for magma being > > > >> available that registers your routine > > > >> > > > >> ierr = > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > > > >> > > > >> Barrt > > > >> > > > >>> > > > >> > > > >>> What else do I need to change if I have to add the functions > > > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm > > > >>> dense matrix multiplication). > > > >>> > > > >>> Harshad > > > >>> > > > >>> > > > >>> ----- Original Message ----- > > > >>> From: "Barry Smith" <[email protected]> > > > >>> To: "Harshad Sahasrabudhe" <[email protected]> > > > >>> Cc: "For users of the development version of PETSc" > > > >>> <[email protected]> > > > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM > > > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > >>> > > > >>> > > > >>> How general do you plan for this "magma" matrix class to be? If it > > > >>> all it is for is to do LU/Cholesky factorizations then you do NOT > > > >>> need to introduce src/mat/impls/dense/seq/magma at all. Simply use > > > >>> the MatGetFactor() mechanism to "How will the user be able to access > > > >>> this function?". This is definitely the easy way to go. > > > >>> > > > >>> If you want to use the "magma" matrix class for all kinds of > > > >>> non-factorization matrix operations then you need to write a full > > > >>> magma matrix class. I don't recommend this. > > > >>> > > > >>> Barry > > > >>> > > > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe > > > >>> <[email protected]> wrote: > > > >>> > > > >>>> Hi, > > > >>>> > > > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I > > > >>>> have created the following files and directories: > > > >>>> > > > >>>> src/mat/impls/dense/seq/magma > > > >>>> src/mat/impls/dense/seq/magma/magma.c > > > >>>> src/mat/impls/dense/seq/magma/makefile > > > >>>> > > > >>>> Right now I am just trying the make LU factorization work through > > > >>>> MAGMA. Barry had suggested looking at dense.c, as the MAGMA function > > > >>>> for LU has the same sequence as the LAPACK function getrf. MAGMA > > > >>>> does the memory allocation on the GPU inside it's function > > > >>>> magma_?getrf. So can I directly use the matrix type declared in > > > >>>> dense.c, and just define a function MatLUFactor_SeqDense_MAGMA > > > >>>> magma.c which uses the same matrix type? > > > >>>> > > > >>>> How will the user be able to access this function? > > > >>>> > > > >>>> Thanks, > > > >>>> Harshad > > > >>> > > > >> > > > > > > > > > > > > >
