On Tue, May 29, 2018 at 4:05 AM, Najeeb Ahmad <[email protected]> wrote:
> > > On Mon, May 28, 2018 at 9:32 PM, Smith, Barry F. <[email protected]> > wrote: > >> >> >> > On May 28, 2018, at 10:32 AM, Najeeb Ahmad <[email protected]> wrote: >> > >> > Thanks a lot Satish for your prompt reply. >> > >> > I just checked that SuperLU_dist package works only for matrices of >> type aij. and uses lu preconditioner. I am currently working with baij >> matrix. What is the best preconditioner choice for baij matrices on >> parallel machines? >> >> The best preconditioner is always problem specific. Where does your >> problem come from? CFD? Structural mechanics? other apps? >> > > I am interested in writing solver for reservoir simulation > employing FVM and unstructured grids. My main objective is to study > performance of the code with different data structures/data layouts and > architecture specific optimizations, specifically targeting the multicore > architectures like KNL for instance. Later the study may be extended to > include GPUs. The options for switching between AIJ and BAIJ etc. are > therefore very useful for my study. > I would strongly encourage you to create a simple performance model first. It should demonstrate that the investigation can produce a positive result. For example, performance for stencil codes like this is usually limited by memory bandwidth. The bandwidth advantage would be at most 3x with KNL/GPU vs Skylake, and it would be nil if the Skylake were to get MCDRAM as we all expect. > The purpose why I wanted to change the preconditioner is that the > default preconditioner is giving me different iterations count for > different number of processesors. I would rather like a preconditioner that > would give me same iteration count for any processor count so that I can > better compare the performance results. > > Your suggestions in this regard are highly appreciated, > specifically with reference to the following points: > > - Is it possible to explicitly use high bandwidth memory in PETSc > for selected object placement (e.g. using memkind library for instance)? > I believe Richard has some extensions for this. Richard? > - What would it take to take advantage of architecture specific > compiler flags to achieve good performance on a given platform (e.g. > -xMIC-AVX512 for AVX512 on KNL, #pragma SIMD etc.). > Usually intrinsics. We already have this for MatMult() for some architectures. Thanks, Matt > > Sorry for some very basic questions as I am a novice PETSc user. > > Thanks for your time :) > >> >> Anyways you probably want to make your code be able to switch >> between AIJ and BAIJ at run time since the different formats support >> somewhat different solvers. If your code alls MatSetFromOptions then you >> can switch via the command line option -mat_type aij or baij >> >> Barry >> >> > >> > Thanks >> > >> > On Mon, May 28, 2018 at 8:23 PM, Satish Balay <[email protected]> >> wrote: >> > On Mon, 28 May 2018, Najeeb Ahmad wrote: >> > >> > > Hi All, >> > > >> > > I have Petsc release version 3.9.2 configured with the following >> options: >> > > >> > > Configure options --with-cc=mpiicc --with-cxx=mpiicpc >> --with-fc=mpiifort >> > > --download-fblaslapack=1 >> > > >> > > Now I want to use PCILU in my code and when I set the PC type to >> PCILU in >> > > the code, I get the following error: >> > > >> > > [0]PETSC ERROR: --------------------- Error Message >> > > -------------------------------------------------------------- >> > > [0]PETSC ERROR: See >> > > http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for >> > > possible LU and Cholesky solvers >> > > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must >> > > ./configure with --download-<package> >> > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html for >> > > trouble shooting. >> > > [0]PETSC ERROR: Petsc Release Version 3.9.2, unknown >> > > [0]PETSC ERROR: ./main on a arch-linux2-c-debug named Karachi by >> nahmad Mon >> > > May 28 17:52:41 2018 >> > > [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >> > > --with-fc=mpiifort --download-fblaslapack=1 >> > > [0]PETSC ERROR: #1 MatGetFactor() line 4318 in >> > > /home/nahmad/PETSc/petsc/src/mat/interface/matrix.c >> > > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in >> > > /home/nahmad/PETSc/petsc/src/ksp/pc/impls/factor/ilu/ilu.c >> > > [0]PETSC ERROR: #3 PCSetUp() line 923 in >> > > /home/nahmad/PETSc/petsc/src/ksp/pc/interface/precon.c >> > > [0]PETSC ERROR: #4 KSPSetUp() line 381 in >> > > /home/nahmad/PETSc/petsc/src/ksp/ksp/interface/itfunc.c >> > > [0]PETSC ERROR: #5 KSPSolve() line 612 in >> > > /home/nahmad/PETSc/petsc/src/ksp/ksp/interface/itfunc.c >> > > [0]PETSC ERROR: #6 SolveSystem() line 60 in >> > > /home/nahmad/Aramco/petsc/petsc/BlockSolveTest/src/main.c >> > > >> > > >> > > I assume that I am missing LU package like SuperLU_dist for instance >> and I >> > > need to download and configure it with Petsc. >> > >> > yes - petsc has sequential LU - but you need superlu_dist/mumps for >> parallel lu. >> > >> > > >> > > I am wondering what is the best way to reconfigure Petsc to download >> and >> > > use the appropriate package to support PCILU? >> > >> > Rerun configure with the additional option --download-superlu_dist=1. >> > >> > You can do this with current PETSC_ARCH you are using [i.e reinstall >> > over the current build] - or use a different PETSC_ARCH - so both >> > builds exist and useable. >> > >> > Satish >> > >> > > >> > > You advice is highly appreciated. >> > > >> > > >> > >> > >> > >> > >> > -- >> > Najeeb Ahmad >> > >> > Research and Teaching Assistant >> > PARallel and MultiCORE Computing Laboratory (ParCoreLab) >> > Computer Science and Engineering >> > Koç University, Istanbul, Turkey >> > >> >> > > > -- > *Najeeb Ahmad* > > > *Research and Teaching Assistant* > *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * > > *Computer Science and Engineering* > *Koç University, Istanbul, Turkey* > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
