I think it should default to true but it definitely should monitor itself, that is measure the convergence for new solves with the old interpolation but new matrix values. It can report this number and (eventually) be self-adaptive.
The problem for self-monitoring is that the PCGAMG is inside the KSP (and thus doesn't know about the behavior of the KSP that is calling it) so we need to introduce a new mechanism to allow a PC to have information about how it is affecting the KSP that calls it. Requires some thought for a good API. Separation of concerns in software is generally a good thing, but this is an example of where having clear separation of concerns between the PC and KSP makes doing a good thing more difficult because in the original KSP/PC design I didn't think about this type of concern. > On Sep 17, 2022, at 10:12 PM, Mark Adams <mfad...@lbl.gov> wrote: > > > > On Sat, Sep 17, 2022 at 5:52 PM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > It is essentially doing AMR each time-step, and for the given application, > I don't think that is pathological. It is an app built on Randy LeVeque's > Clawpack stuff. The linear solver totally dominates the time which makes > users of Clawpack very hesitant to consider methods with implicit steps, even > when they need them. One linear solution takes far longer than the entire > explicit step including its sub-grid cycling and the adaptive changes to the > grid at each time-step. > > There is sub-grid cycling where similar solves are done on the same mesh > (hence same nonzero pattern) two or three times, so thanks, we will try > -pc_gamg_reuse_interpolation true and it could potentially help a good amount. > > There could be a mode where -pc_gamg_reuse_interpolation true is on by > default, and the KSP/PC monitors the performance (convergence rate) of the > solve following the reuse to decide if sticking with the old interpolation is > ok or if a new interpolation should be done for the next change in the matrix > values. Thus not requiring user knowledge and tuning of this option which > most users who just want to get on with their work would not want to mess > with. > > I don't want to get complicated, just look at the default. > One could give the user finer grain control over the scheduling, but I just > think it would confuse users and would be hard to understand. > I could see, in theory making this a lag integer instead of a bool, but this > has never come up (see below) so it is not worth the API churn now. > > My original thinking was that since AMG adapts to the matrix it makes sense > to redo the space each time mathematically, but after thinking about it and > living with it, the default should be TRUE. > I've never seen a problem that changes so much with time that the coarse grid > space is improved dramatically from reconstructing the spaces/aggregates. > If we get a user that says AMG convergence is slowing down during a run for > no discernible reason then tell them to use FALSE and see if convergence > improves. > AMG coarsening is really not very precise anyway, based on heuristics that > are easily defeated, so it is just overkill to redo it. > If a user wants to add some heuristic then they can just delete/reset the > solver. I don't think there is any significant overhead from doing that. > (ie, we could just remove this parameter always reuse and let the user reset > manually and maybe that is a good idea now that I think about it) > > I use TRUE in all examples, in the hope that users will copy my input decks, > but that did not happen here. > I have looked over my other recommended parameters and none of them should > obviously have a different default. > > I have added this to my current MR but I am now thinking that it might be > better to remove this flag and just tell users to reset the KSP/PC if they > want to redo it. > This has bugged me for a while and this user just reminded me and I have a > little cleanup MR going.... > > What do you think about just removing this and always reuse? > > Mark > > > > > > > >> On Sep 17, 2022, at 1:43 PM, Mark Adams <mfad...@lbl.gov >> <mailto:mfad...@lbl.gov>> wrote: >> >> I don't see a problem here other than the network looks bad relative to the >> problem size. >> >> All the graph methods (PCGAMGCreateG and MIS) are 2x slower. >> - THe symmetrization must be in PCGAMGCreateG. >> - MIS is pretty old code (the algorithm and original code are 25 years old) >> RAPs are about the same. >> KSPGMRESOrthog and MatMult are nowhere near perfect. >> >> The graph setup work gets amortized by (most) applications and benchmarkers >> that know how to benchmark, so it is not highly engineered like the RAP and >> MatMult. >> Note, this application is building the graph work for every linear solve. >> I am guessing they want '-pc_gamg_reuse_interpolation true' or are doing a >> single step/stage TS with a linear problem and AMR every time step, which >> would be pretty pathological. >> I'm doing an MR right now, maybe I should change the default for >> -pc_gamg_reuse_interpolation? >> >> Mark >> >> >> >> On Sat, Sep 17, 2022 at 10:12 AM Barry Smith <bsm...@petsc.dev >> <mailto:bsm...@petsc.dev>> wrote: >> >> Sure, but have you ever seen such a large jump in time in going from one >> to two MPI ranks, and are there any algorithms to do the aggregation that >> would not require this very expensive parallel symmetrization? >> >>> On Sep 17, 2022, at 9:07 AM, Mark Adams <mfad...@lbl.gov >>> <mailto:mfad...@lbl.gov>> wrote: >>> >>> Symetrix graph make a transpose and then adds them. >>> I imagine adding two different matrices is expensive. >>> >>> On Fri, Sep 16, 2022 at 8:30 PM Barry Smith <bsm...@petsc.dev >>> <mailto:bsm...@petsc.dev>> wrote: >>> >>> Mark, >>> >>> I have runs of GAMG on one and two ranks with -pc_gamg_symmetrize_graph >>> because the matrix is far from symmetric and some of GAMG is taking a huge >>> amount more time with 2 ranks than one. (While other stuff like VecNorm >>> shows improvement with two ranks). I've attached the two files >>> >>> Have you seen this before, is there anything that can be done about? If >>> going to two ranks causes almost a doubling in GAMG setup time that makes >>> using parallelism not useful, >>> >>> Barry >>> >> >
