I'd like to get an idea of the solver performance when using mpibaij. Using my existing code based on mpiaij I simply swapped the preallocation routine and switched to mpibaij using the runtime flag. I basically perform the following function calls:
MatCreate(PETSC_COMM_WORLD, &SystMat); MatSetSizes(SystMat, PETSC_DECIDE, PETSC_DECIDE, nDOFg, nDOFg); MatSetFromOptions(SystMat); MatMPIBAIJSetPreallocation(SystMat, 5, 30, PETSC_NULL,10, PETSC_NULL); VecCreate(PETSC_COMM_WORLD, &Rhs); VecSetSizes(Rhs, PETSC_DECIDE, nDOFg); VecSetFromOptions(Rhs); /* Filling SystMat and Rhs (currently using MatSetValues() ) */ ... VecDuplicate(*Rhs, &Sol); KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetOperators(ksp, *SystMat, *SystMat, DIFFERENT_NONZERO_PATTERN); KSPSetFromOptions(ksp); KSPSolve(ksp, *Rhs, Sol); My runtime options are: -mat_type mpibaij -ksp_type lgmres -ksp_rtol 1.e-8 -ksp_gmres_restart 200 -pc_type asm -log_summary This is what I get using 2 processors: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Vector wrong size 34178 for scatter 34180 (scatter forward and vector from != ctx from size)! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 [0]PETSC ERROR: Libraries linked from ... [0]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 [0]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c [0]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c [0]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Nonconforming object sizes! [1]PETSC ERROR: Vector wrong size 34177 for scatter 34175 (scatter forward and vector from != ctx from size)! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 [1]PETSC ERROR: Libraries linked from ... [1]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 [1]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c [1]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c [1]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c Is there a basic mistake in my procedure? Thanks Henning
