Please provide a bit more detail about the operator. Is it a pressure solver for CFD? Cell centered? Does the matrix has a null space of the constant functions? Is it the same linear system for each "time-step?" in the CFD solver or different?
How many iterations is hypre BoomerAMG typically taking? This is an extremely tight tolerance, why do you set it so small? KSPSetTolerances(ksp, 1e-13, 1e-13, PETSC_DEFAULT, PETSC_DEFAULT) ; Send the output from a run with -log_summary so we can see where the time is being spent. Barry > On Jul 8, 2015, at 11:11 AM, Ganesh Vijayakumar <[email protected]> wrote: > > Hello, > > First of all..thanks to the PETSC developers and everyone else contributing > supporting material on the web. > > I need to solve a system of equations Ax =b, where A is symmetric, sparse, > unstructured and in parallel as a part of a finite volume solver for CFD. > Decomposition is already done. I initialize the matrix as MPIAIJ even though > it's symmetric as that's the only thing I could get to work. > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetType(A,MATMPIAIJ); > MatSetSizes(A,nCellsCurProc,nCellsCurProc,nTotalCells,nTotalCells); > > MatMPIAIJSetPreallocation(A,10,PETSC_NULL,5,PETSC_NULL); > > MatSetUp(A) ; > // Assemble matrix using MatSetValue.. assemble full matrix even though it's > symmetric. > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY) ; > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY) ; > //Create solver > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetType(ksp,KSPFGMRES;) > > KSPGetPC(ksp, &pc); > PCFactorSetShiftType(pc, MAT_SHIFT_POSITIVE_DEFINITE); > > //PCSetType(pc, PCML); //Trilinos - Multilevel > > //PCSetType(pc, PCBJACOBI) ; //Incomplete LU > //PetscOptionsSetValue("-sub_pc_type", "ilu") ; > KSPSetFromOptions(ksp); > > > // PCSetType(pc, PCASM); //Additive Schwarz Methods > // PCASMSetOverlap(pc,overlap); > > PCSetType(pc, PCHYPRE); //HYPRE > PCHYPRESetType(pc, "boomeramg"); > > //PCHYPRESetType(pc, "parasails"); > > KSPSetPC(ksp, pc); > KSPSetTolerances(ksp, 1e-13, 1e-13, PETSC_DEFAULT, PETSC_DEFAULT) ; > KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); > KSPSetOperators(ksp, A, A); > KSPMonitorDefault(ksp, 1, 1, pVoid) ; > > KSPSolve(ksp,b,x); > VecGetArray(x, &getResultArray) ; > KSPGetResidualNorm(ksp, &endNorm) ; > KSPGetIterationNumber(ksp, &nIterations); > > KSPDestroy(&ksp); > > > I know that this works and gives me the correct result. However, boomerAMG > is proving to be too slow.. atleast the way I use it at 256 processors with > around 12 million unknowns. I need your advice on the preconditioner. Am I > using it right? Is there anything else I could I be doing better? > > ganesh
