On Mon, Sep 29, 2014 at 3:47 PM, Filippo Leonardi < [email protected]> wrote:
> @ Barry: It may be that I forgot to set the number of levels for the runs. > > New experiment with the following options: > > -da_refine 5 -pc_type mg -ksp_monitor -log_summary -pc_mg_type full > -ksp_view - > pc_mg_log -pc_mg_levels 5 -pc_mg_galerkin -ksp_monitor_true_residual - > ksp_converged_reason > > on 128^3, and it looks nice: > > 0 KSP Residual norm 5.584601494955e+01 > 0 KSP preconditioned resid norm 5.584601494955e+01 true resid norm > 1.370259979011e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP Residual norm 9.235021247277e+00 > 1 KSP preconditioned resid norm 9.235021247277e+00 true resid norm > 8.185195475443e-01 ||r(i)||/||b|| 5.973461679404e-02 > 2 KSP Residual norm 6.344253555076e-02 > 2 KSP preconditioned resid norm 6.344253555076e-02 true resid norm > 1.108015805992e-01 ||r(i)||/||b|| 8.086172134956e-03 > 3 KSP Residual norm 1.084530268454e-03 > 3 KSP preconditioned resid norm 1.084530268454e-03 true resid norm > 3.228589340041e-03 ||r(i)||/||b|| 2.356187431214e-04 > 4 KSP Residual norm 2.345341850850e-05 > 4 KSP preconditioned resid norm 2.345341850850e-05 true resid norm > 9.362117433445e-05 ||r(i)||/||b|| 6.832365811489e-06 > Linear solve converged due to CONVERGED_RTOL iterations 4 > > I'll try on more processors. Is it correct what I am doing? If so, is there > some tweak I am missing. > > Any suggestion on optimal number of levels VS number of processors? > > Btw, thanks a lot, you are always so helpful. > > On Monday 29 September 2014 14:59:49 you wrote: > > Filippo Leonardi <[email protected]> writes: > > > Thank you. > > > > > > Actually I had the feeling that it wasn't my problem with Bjacobi and > CG. > > > > > > So I'll stick to MG. Problem with MG is that there are a lot of > parameters > > > to be tuned, so I leave the defaults (expect I select CG as Krylow > > > method). I post just results for 64^3 and 128^3. Tell me if I'm missing > > > some useful detail. (I get similar results with BoomerAMG). > > > > > > Time for one KSP iteration (-ksp_type cg -log_summary -pc_mg_galerkin > > > -pc_type mg): > > > 32^3 and 1 proc: 1.01e-1 > > > 64^3 and 8 proc: 6.56e-01 > > > 128^3 and 64 proc: 1.05e+00 > > > Number of PCSetup per KSPSolve: > > > 15 > > > 39 > > > 65 > > > > Presumably you mean PCApply. Something is wrong here because this > > iteration count is way too high. Perhaps your boundary conditions are > > nonsymmetric or interpolation is not compatible with the discretization. > > > > > With BoomerAMG: > > > stable 8 iterations per KSP but time per iteration greater than PETSc > MG > > > and still increases: > > > 64^3: 3.17e+00 > > > 128^3: 9.99e+00 > > > > > --> For instance with 64^3 (256 iterations): > > In the first pass with geometric multigrid, don't worry about timing and > > get the iterations figured out. Are you using a cell-centered or > > vertex-centered discretization. When you say 128^3, is that counting > > the number of elements or the number of vertices? Note that if you have > > a vertex-centered discretization, you will want a 129^3 grid. > > Cell-centered, counting elements. > > > With > > PCMG, make sure you are getting the number of levels of refinement that > > you expect. > > > > > > > You should see something like the following (this is 193^3). > > > > $ mpiexec -n 4 ./ex45 -da_refine 5 -pc_type mg -ksp_monitor -pc_mg_type > full > > -mg_levels_ksp_type richardson -mg_levels_pc_type sor -ksp_type > richardson > > 0 KSP Residual norm 2.653722249919e+03 > > 1 KSP Residual norm 1.019366121923e+02 > > 2 KSP Residual norm 2.364558296616e-01 > > 3 KSP Residual norm 7.438761746501e-04 > > Residual norm 1.47939e-06 > > > > > You can actually do better than this by using higher order FMG > > interpolation, by going matrix-free, etc. For example, HPGMG > > (finite-element or finite-volume, see https://hpgmg.org) will solve more > > than a million equations/second per core. Is your application really > > solving the constant-coefficient Poisson problem on a Cartesian grid, or > > is that just a test? > > I actually just need a cell centered Poisson solver on cartesian grids. > (various boundary conditions). > > Matrix free you mean AMG (like -pc_mg_galerkin)? Does it reach the same > scalability as GMG? No, Jed means calculating the action of the operator matrix-free instead of using an assembled sparse matrix. > > > > > > Using Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT > 2012 > > > > And a reminder to please upgrade to the current version of PETSc. > > Sadly this is not in my power. I had actually had to rollback all the APIs > to > be able to do this test runs. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
