Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
On Fri, Oct 26, 2018 at 12:13 PM Jed Brown wrote:
> Matthew Knepley writes:
>
> > On Fri, Oct 26, 2018 at 11:24 AM Jed Brown wrote:
> >
> >> Matthew Knepley writes:
> >>
> >> > On Fri, Oct 26, 2018 at 11:06 AM Jed Brown wrote:
> >> >
> >> >> Matthew Knepley writes:
> >> >>
> >> >> > I am having a problem, and every solution I can think of would seem
> >> to do
> >> >> > violence to encapsulation or to DMKSP/DMSNES.
> >> >> >
> >> >> > I want to have GMG automatically work on the velocity part of
> Stokes.
> >> So
> >> >> I
> >> >> > give
> >> >> >
> >> >> > -fieldsplit_velocity_pc_type mg
> >> >> >
> >> >> > in SNES ex62. This complains that the KSP has a DM but
> >> >> > KSPSetComputeOperator() has not been called. It is fine on the
> global
> >> >> > system because SNES called that in SNESSetupMatrices(). Thus in
> >> >> FieldSplit,
> >> >> > I inserted
> >> >> >
> >> >> > {
> >> >> > PetscErrorCode (*func)(KSP,Mat,Mat,void*);
> >> >> > void*ctx;
> >> >> >
> >> >> > ierr = DMKSPGetComputeOperators(pc->dm, ,
> >> >> > );CHKERRQ(ierr);
> >> >> > ierr = DMKSPSetComputeOperators(dms[i], func,
> >> >> > ctx);CHKERRQ(ierr);
> >> >> > }
> >> >>
> >> >> Wait, pc->dm is the coupled system while each dms[i] is a component
> >> >> (like velocity). It would be awkward to ask the user to write a
> single
> >> >> function that would inspect the DM to figure out what sort of
> problem it
> >> >> was discretizing (velocity-only, pressure-only,
> >> >> velocity-pressure-coupled, etc.). Do you currently have a
> >> >> ComputeOperators for each sub-problem?
> >> >>
> >> >
> >> > CreateSubDM() returns a DM whose DS only involves the subset of
> fields.
> >>
> >> DS? That isn't really a DM-level concept; I see you use it in a couple
> >> utility functions in DMDA, but it isn't part of the solver-related
> >> functionality. In any case, the code you wrote above looks to be moving
> >> the outer ComputeOperators onto a sub-DM, which can't possibly be a
> >> desirable solution. Why can't DMCreateSubDM ensure that each sub DM is
> >> endowed with whatever discretization functionality is known?
> >>
> >
> > As I said in my reply, it already does this. HOWEVER, it cannot move the
> > construction functions from the old DMSNES to the subDMSNES, because it
> > does not know about SNES.
>
> DMSNES could register a callback to decompose itself for DMCreateSubDM.
> We do something similar in DMRestrictHook, for example.
>
It looks like we will have to add a hook in order to break dependency
cycles. I will
do it.
Matt
> Alternatively, the user could create the subDMs and endow them with
> functionality. You'd probably want DMGetSubDM for that since the parent
> DM would maintain ownership.
>
> >> > Thus, only
> >> > the v-v block is present in the subDM, and when the SNES assembly
> >> functions
> >> > is called,
> >> > it forms that block. I just need that function to be called.
> >>
> >> Is it really the same function as is used by the outer DM?
> >
> >
> > Yes.
> >
> > You are not addressing the question.
>
> I just wanted to clarify whether you thought that was something that
> could possibly be merged or just an attempted dirty hack.
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
Matthew Knepley writes:
> On Fri, Oct 26, 2018 at 11:24 AM Jed Brown wrote:
>
>> Matthew Knepley writes:
>>
>> > On Fri, Oct 26, 2018 at 11:06 AM Jed Brown wrote:
>> >
>> >> Matthew Knepley writes:
>> >>
>> >> > I am having a problem, and every solution I can think of would seem
>> to do
>> >> > violence to encapsulation or to DMKSP/DMSNES.
>> >> >
>> >> > I want to have GMG automatically work on the velocity part of Stokes.
>> So
>> >> I
>> >> > give
>> >> >
>> >> > -fieldsplit_velocity_pc_type mg
>> >> >
>> >> > in SNES ex62. This complains that the KSP has a DM but
>> >> > KSPSetComputeOperator() has not been called. It is fine on the global
>> >> > system because SNES called that in SNESSetupMatrices(). Thus in
>> >> FieldSplit,
>> >> > I inserted
>> >> >
>> >> > {
>> >> > PetscErrorCode (*func)(KSP,Mat,Mat,void*);
>> >> > void*ctx;
>> >> >
>> >> > ierr = DMKSPGetComputeOperators(pc->dm, ,
>> >> > );CHKERRQ(ierr);
>> >> > ierr = DMKSPSetComputeOperators(dms[i], func,
>> >> > ctx);CHKERRQ(ierr);
>> >> > }
>> >>
>> >> Wait, pc->dm is the coupled system while each dms[i] is a component
>> >> (like velocity). It would be awkward to ask the user to write a single
>> >> function that would inspect the DM to figure out what sort of problem it
>> >> was discretizing (velocity-only, pressure-only,
>> >> velocity-pressure-coupled, etc.). Do you currently have a
>> >> ComputeOperators for each sub-problem?
>> >>
>> >
>> > CreateSubDM() returns a DM whose DS only involves the subset of fields.
>>
>> DS? That isn't really a DM-level concept; I see you use it in a couple
>> utility functions in DMDA, but it isn't part of the solver-related
>> functionality. In any case, the code you wrote above looks to be moving
>> the outer ComputeOperators onto a sub-DM, which can't possibly be a
>> desirable solution. Why can't DMCreateSubDM ensure that each sub DM is
>> endowed with whatever discretization functionality is known?
>>
>
> As I said in my reply, it already does this. HOWEVER, it cannot move the
> construction functions from the old DMSNES to the subDMSNES, because it
> does not know about SNES.
DMSNES could register a callback to decompose itself for DMCreateSubDM.
We do something similar in DMRestrictHook, for example.
Alternatively, the user could create the subDMs and endow them with
functionality. You'd probably want DMGetSubDM for that since the parent
DM would maintain ownership.
>> > Thus, only
>> > the v-v block is present in the subDM, and when the SNES assembly
>> functions
>> > is called,
>> > it forms that block. I just need that function to be called.
>>
>> Is it really the same function as is used by the outer DM?
>
>
> Yes.
>
> You are not addressing the question.
I just wanted to clarify whether you thought that was something that
could possibly be merged or just an attempted dirty hack.
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
On Fri, Oct 26, 2018 at 11:24 AM Jed Brown wrote:
> Matthew Knepley writes:
>
> > On Fri, Oct 26, 2018 at 11:06 AM Jed Brown wrote:
> >
> >> Matthew Knepley writes:
> >>
> >> > I am having a problem, and every solution I can think of would seem
> to do
> >> > violence to encapsulation or to DMKSP/DMSNES.
> >> >
> >> > I want to have GMG automatically work on the velocity part of Stokes.
> So
> >> I
> >> > give
> >> >
> >> > -fieldsplit_velocity_pc_type mg
> >> >
> >> > in SNES ex62. This complains that the KSP has a DM but
> >> > KSPSetComputeOperator() has not been called. It is fine on the global
> >> > system because SNES called that in SNESSetupMatrices(). Thus in
> >> FieldSplit,
> >> > I inserted
> >> >
> >> > {
> >> > PetscErrorCode (*func)(KSP,Mat,Mat,void*);
> >> > void*ctx;
> >> >
> >> > ierr = DMKSPGetComputeOperators(pc->dm, ,
> >> > );CHKERRQ(ierr);
> >> > ierr = DMKSPSetComputeOperators(dms[i], func,
> >> > ctx);CHKERRQ(ierr);
> >> > }
> >>
> >> Wait, pc->dm is the coupled system while each dms[i] is a component
> >> (like velocity). It would be awkward to ask the user to write a single
> >> function that would inspect the DM to figure out what sort of problem it
> >> was discretizing (velocity-only, pressure-only,
> >> velocity-pressure-coupled, etc.). Do you currently have a
> >> ComputeOperators for each sub-problem?
> >>
> >
> > CreateSubDM() returns a DM whose DS only involves the subset of fields.
>
> DS? That isn't really a DM-level concept; I see you use it in a couple
> utility functions in DMDA, but it isn't part of the solver-related
> functionality. In any case, the code you wrote above looks to be moving
> the outer ComputeOperators onto a sub-DM, which can't possibly be a
> desirable solution. Why can't DMCreateSubDM ensure that each sub DM is
> endowed with whatever discretization functionality is known?
>
As I said in my reply, it already does this. HOWEVER, it cannot move the
construction functions from the old DMSNES to the subDMSNES, because it
does not know about SNES.
> > Thus, only
> > the v-v block is present in the subDM, and when the SNES assembly
> functions
> > is called,
> > it forms that block. I just need that function to be called.
>
> Is it really the same function as is used by the outer DM?
Yes.
You are not addressing the question.
Matt
> That is not
> a general-purpose solution because users can set
> DMKSPSetComputeOperators/KSPSetComputeOperator to directly call their
> own functions.
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
Matthew Knepley writes:
> On Fri, Oct 26, 2018 at 11:06 AM Jed Brown wrote:
>
>> Matthew Knepley writes:
>>
>> > I am having a problem, and every solution I can think of would seem to do
>> > violence to encapsulation or to DMKSP/DMSNES.
>> >
>> > I want to have GMG automatically work on the velocity part of Stokes. So
>> I
>> > give
>> >
>> > -fieldsplit_velocity_pc_type mg
>> >
>> > in SNES ex62. This complains that the KSP has a DM but
>> > KSPSetComputeOperator() has not been called. It is fine on the global
>> > system because SNES called that in SNESSetupMatrices(). Thus in
>> FieldSplit,
>> > I inserted
>> >
>> > {
>> > PetscErrorCode (*func)(KSP,Mat,Mat,void*);
>> > void*ctx;
>> >
>> > ierr = DMKSPGetComputeOperators(pc->dm, ,
>> > );CHKERRQ(ierr);
>> > ierr = DMKSPSetComputeOperators(dms[i], func,
>> > ctx);CHKERRQ(ierr);
>> > }
>>
>> Wait, pc->dm is the coupled system while each dms[i] is a component
>> (like velocity). It would be awkward to ask the user to write a single
>> function that would inspect the DM to figure out what sort of problem it
>> was discretizing (velocity-only, pressure-only,
>> velocity-pressure-coupled, etc.). Do you currently have a
>> ComputeOperators for each sub-problem?
>>
>
> CreateSubDM() returns a DM whose DS only involves the subset of fields.
DS? That isn't really a DM-level concept; I see you use it in a couple
utility functions in DMDA, but it isn't part of the solver-related
functionality. In any case, the code you wrote above looks to be moving
the outer ComputeOperators onto a sub-DM, which can't possibly be a
desirable solution. Why can't DMCreateSubDM ensure that each sub DM is
endowed with whatever discretization functionality is known?
> Thus, only
> the v-v block is present in the subDM, and when the SNES assembly functions
> is called,
> it forms that block. I just need that function to be called.
Is it really the same function as is used by the outer DM? That is not
a general-purpose solution because users can set
DMKSPSetComputeOperators/KSPSetComputeOperator to directly call their
own functions.
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
Hi, I'm just wondering if this ties into the issues I brought up in :
https://lists.mcs.anl.gov/pipermail/petsc-dev/2018-October/023615.html ?
>From my perspective, I think I would prefer the copy everything mechanism,
but I'm not sure about the nature of the introduced dependencies and if I
would have to make extra adjustments from my end.
Thanks for any clarification,
- Boris
On Fri, Oct 26, 2018 at 10:49 AM Matthew Knepley wrote:
> I am having a problem, and every solution I can think of would seem to do
> violence to encapsulation or to DMKSP/DMSNES.
>
> I want to have GMG automatically work on the velocity part of Stokes. So I
> give
>
> -fieldsplit_velocity_pc_type mg
>
> in SNES ex62. This complains that the KSP has a DM but
> KSPSetComputeOperator() has not been called. It is fine on the global
> system because SNES called that in SNESSetupMatrices(). Thus in FieldSplit,
> I inserted
>
> {
> PetscErrorCode (*func)(KSP,Mat,Mat,void*);
> void*ctx;
>
> ierr = DMKSPGetComputeOperators(pc->dm, ,
> );CHKERRQ(ierr);
> ierr = DMKSPSetComputeOperators(dms[i], func,
> ctx);CHKERRQ(ierr);
> }
>
> where we call KSPSetDM(). However, this does not work, because the subDM
> has a new DMSNES, which does not have the information about the problem.
>
> Solution #1:
>
> Copy the info from the old DMSNES to the subDMSNES. This inserts a
> dependence between KSP and SNES which we do not want.
>
> Solution #2:
>
> Have DMCreateSubDM() copy everything from the solvers. This also inserts
> dependencies, unless we have a "copy everything" mechanism, but is this
> what we want.
>
> Solution #3:
>
> Rethink DMKSP/DMSNES. Is this how we should store problem information.
> Dave already had another problem with the organization.
>
> What should be done? I have never understood this construct very well.
> Right now, this is holding up doing some computations I need.
>
> Thanks,
>
> Matt
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
On Fri, Oct 26, 2018 at 11:06 AM Jed Brown wrote:
> Matthew Knepley writes:
>
> > I am having a problem, and every solution I can think of would seem to do
> > violence to encapsulation or to DMKSP/DMSNES.
> >
> > I want to have GMG automatically work on the velocity part of Stokes. So
> I
> > give
> >
> > -fieldsplit_velocity_pc_type mg
> >
> > in SNES ex62. This complains that the KSP has a DM but
> > KSPSetComputeOperator() has not been called. It is fine on the global
> > system because SNES called that in SNESSetupMatrices(). Thus in
> FieldSplit,
> > I inserted
> >
> > {
> > PetscErrorCode (*func)(KSP,Mat,Mat,void*);
> > void*ctx;
> >
> > ierr = DMKSPGetComputeOperators(pc->dm, ,
> > );CHKERRQ(ierr);
> > ierr = DMKSPSetComputeOperators(dms[i], func,
> > ctx);CHKERRQ(ierr);
> > }
>
> Wait, pc->dm is the coupled system while each dms[i] is a component
> (like velocity). It would be awkward to ask the user to write a single
> function that would inspect the DM to figure out what sort of problem it
> was discretizing (velocity-only, pressure-only,
> velocity-pressure-coupled, etc.). Do you currently have a
> ComputeOperators for each sub-problem?
>
CreateSubDM() returns a DM whose DS only involves the subset of fields.
Thus, only
the v-v block is present in the subDM, and when the SNES assembly functions
is called,
it forms that block. I just need that function to be called.
Matt
> > where we call KSPSetDM(). However, this does not work, because the subDM
> > has a new DMSNES, which does not have the information about the problem.
> >
> > Solution #1:
> >
> > Copy the info from the old DMSNES to the subDMSNES. This inserts a
> > dependence between KSP and SNES which we do not want.
> >
> > Solution #2:
> >
> > Have DMCreateSubDM() copy everything from the solvers. This also
> inserts
> > dependencies, unless we have a "copy everything" mechanism, but is this
> > what we want.
> >
> > Solution #3:
> >
> > Rethink DMKSP/DMSNES. Is this how we should store problem information.
> > Dave already had another problem with the organization.
> >
> > What should be done? I have never understood this construct very well.
> > Right now, this is holding up doing some computations I need.
> >
> > Thanks,
> >
> > Matt
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
Re: [petsc-dev] Problem with DMKSP/DMSNES and GMG
Matthew Knepley writes:
> I am having a problem, and every solution I can think of would seem to do
> violence to encapsulation or to DMKSP/DMSNES.
>
> I want to have GMG automatically work on the velocity part of Stokes. So I
> give
>
> -fieldsplit_velocity_pc_type mg
>
> in SNES ex62. This complains that the KSP has a DM but
> KSPSetComputeOperator() has not been called. It is fine on the global
> system because SNES called that in SNESSetupMatrices(). Thus in FieldSplit,
> I inserted
>
> {
> PetscErrorCode (*func)(KSP,Mat,Mat,void*);
> void*ctx;
>
> ierr = DMKSPGetComputeOperators(pc->dm, ,
> );CHKERRQ(ierr);
> ierr = DMKSPSetComputeOperators(dms[i], func,
> ctx);CHKERRQ(ierr);
> }
Wait, pc->dm is the coupled system while each dms[i] is a component
(like velocity). It would be awkward to ask the user to write a single
function that would inspect the DM to figure out what sort of problem it
was discretizing (velocity-only, pressure-only,
velocity-pressure-coupled, etc.). Do you currently have a
ComputeOperators for each sub-problem?
> where we call KSPSetDM(). However, this does not work, because the subDM
> has a new DMSNES, which does not have the information about the problem.
>
> Solution #1:
>
> Copy the info from the old DMSNES to the subDMSNES. This inserts a
> dependence between KSP and SNES which we do not want.
>
> Solution #2:
>
> Have DMCreateSubDM() copy everything from the solvers. This also inserts
> dependencies, unless we have a "copy everything" mechanism, but is this
> what we want.
>
> Solution #3:
>
> Rethink DMKSP/DMSNES. Is this how we should store problem information.
> Dave already had another problem with the organization.
>
> What should be done? I have never understood this construct very well.
> Right now, this is holding up doing some computations I need.
>
> Thanks,
>
> Matt
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
