The code I am using here is the example 42 of PETSc (https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html). Indeed it solves the Stokes equation. I thought it was a good idea to use an example you might know (and didn't find any that uses GAMG functions). I just changed the PCMG setup so that the memory problem appears. And it appears when adding PCGAMG.
I don't care about the performance or even the result rightness here, but only about the difference in memory use between 3.6 and 3.10. Do you think finding a more adapted script would help? I used the threshold of 0.1 only once, at the beginning, to test its influence. I used the default threshold (of 0, I guess) for all the other runs. Myriam Le 03/11/19 à 13:52, Mark Adams a écrit : > In looking at this larger scale run ... > > * Your eigen estimates are much lower than your tiny test problem. > But this is Stokes apparently and it should not work anyway. Maybe you > have a small time step that adds a lot of mass that brings the eigen > estimates down. And your min eigenvalue (not used) is positive. I > would expect negative for Stokes ... > > * You seem to be setting a threshold value of 0.1 -- that is very high > > * v3.6 says "using nonzero initial guess" but this is not in v3.10. > Maybe we just stopped printing that. > > * There were some changes to coasening parameters in going from v3.6 > but it does not look like your problem was effected. (The coarsening > algo is non-deterministic by default and you can see small difference > on different runs) > > * We may have also added a "noisy" RHS for eigen estimates by default > from v3.6. > > * And for non-symetric problems you can try -pc_gamg_agg_nsmooths 0, > but again GAMG is not built for Stokes anyway. > > > On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette > <[email protected] <mailto:[email protected]>> wrote: > > I used PCView to display the size of the linear system in each > level of the MG. You'll find the outputs attached to this mail > (zip file) for both the default threshold value and a value of > 0.1, and for both 3.6 and 3.10 PETSc versions. > > For convenience, I summarized the information in a graph, also > attached (png file). > > As you can see, there are slight differences between the two > versions but none is critical, in my opinion. Do you see anything > suspicious in the outputs? > > + I can't find the default threshold value. Do you know where I > can find it? > > Thanks for the follow-up > > Myriam > > > Le 03/05/19 à 14:06, Matthew Knepley a écrit : >> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Matt, >> >> I plotted the memory scalings using different threshold >> values. The two scalings are slightly translated (from -22 to >> -88 mB) but this gain is neglectable. The 3.6-scaling keeps >> being robust while the 3.10-scaling deteriorates. >> >> Do you have any other suggestion? >> >> Mark, what is the option she can give to output all the GAMG data? >> >> Also, run using -ksp_view. GAMG will report all the sizes of its >> grids, so it should be easy to see >> if the coarse grid sizes are increasing, and also what the effect >> of the threshold value is. >> >> Thanks, >> >> Matt >> >> Thanks >> >> Myriam >> >> Le 03/02/19 à 02:27, Matthew Knepley a écrit : >>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via >>> petsc-users <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi, >>> >>> I used to run my code with PETSc 3.6. Since I upgraded >>> the PETSc version >>> to 3.10, this code has a bad memory scaling. >>> >>> To report this issue, I took the PETSc script ex42.c and >>> slightly >>> modified it so that the KSP and PC configurations are >>> the same as in my >>> code. In particular, I use a "personnalised" multi-grid >>> method. The >>> modifications are indicated by the keyword "TopBridge" >>> in the attached >>> scripts. >>> >>> To plot the memory (weak) scaling, I ran four >>> calculations for each >>> script with increasing problem sizes and computations cores: >>> >>> 1. 100,000 elts on 4 cores >>> 2. 1 million elts on 40 cores >>> 3. 10 millions elts on 400 cores >>> 4. 100 millions elts on 4,000 cores >>> >>> The resulting graph is also attached. The scaling using >>> PETSc 3.10 >>> clearly deteriorates for large cases, while the one >>> using PETSc 3.6 is >>> robust. >>> >>> After a few tests, I found that the scaling is mostly >>> sensitive to the >>> use of the AMG method for the coarse grid (line 1780 in >>> main_ex42_petsc36.cc). In particular, the performance >>> strongly >>> deteriorates when commenting lines 1777 to 1790 (in >>> main_ex42_petsc36.cc). >>> >>> Do you have any idea of what changed between version 3.6 >>> and version >>> 3.10 that may imply such degradation? >>> >>> >>> I believe the default values for PCGAMG changed between >>> versions. It sounds like the coarsening rate >>> is not great enough, so that these grids are too large. This >>> can be set using: >>> >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >>> >>> There is some explanation of this effect on that page. Let >>> us know if setting this does not correct the situation. >>> >>> Thanks, >>> >>> Matt >>> >>> >>> Let me know if you need further information. >>> >>> Best, >>> >>> Myriam Peyrounette >>> >>> >>> -- >>> Myriam Peyrounette >>> CNRS/IDRIS - HLST >>> -- >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/%7Eknepley/> >> >> -- >> Myriam Peyrounette >> CNRS/IDRIS - HLST >> -- >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/%7Eknepley/> > > -- > Myriam Peyrounette > CNRS/IDRIS - HLST > -- > -- Myriam Peyrounette CNRS/IDRIS - HLST --
smime.p7s
Description: Signature cryptographique S/MIME
