On Mon, Feb 10, 2020 at 5:08 AM Dave May <[email protected]> wrote:
> > > On Mon 10. Feb 2020 at 13:09, Matthew Knepley <[email protected]> wrote: > >> On Sun, Feb 9, 2020 at 11:11 PM Pierre Jolivet < >> [email protected]> wrote: >> >>> >>> >>> On 10 Feb 2020, at 6:20 AM, Matthew Knepley <[email protected]> wrote: >>> >>> On Sun, Feb 9, 2020 at 3:23 PM Pierre Jolivet < >>> [email protected]> wrote: >>> >>>> Hello, >>>> I’ve a hard time answering the following DMPlex questions by just >>>> looking at some of the examples and manual. >>>> Considering two DMPlex dm and dma, as in >>>> petsc/src/dm/impls/plex/examples/tests/ex19.c, I’d like to interpolate a >>>> simple P_1 FE function from dm to dma. >>>> The DMCreateInterpolation call gives me: >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Number of fine indices 0 != 4 dual basis vecs >>>> […] >>>> >>> >>> It looks like your fine grid has no discretization, since 0 is >>> numFIndices from >>> >>> ierr = DMPlexGetClosureIndices(dmf, fsection, globalFSection, cell, >>> &numFIndices, &findices, NULL);CHKERRQ(ierr); >>> >>> >>>> [0]PETSC ERROR: #1 DMPlexComputeInterpolatorGeneral() line 2508 in >>>> petsc/src/dm/impls/plex/plexfem.c >>>> [0]PETSC ERROR: #2 DMCreateInterpolation_Plex() line 7688 in >>>> petsc/src/dm/impls/plex/plex.c >>>> [0]PETSC ERROR: #3 DMCreateInterpolation() line 1139 in >>>> petsc/src/dm/interface/dm.c >>>> But the DMs look OK, don’t they, cf. below? >>>> So I have three simple questions: >>>> 1) are all tests at the bottom of ex19.c broken because of PRAgMaTIc or >>>> because of DMPlex currently not supporting some operations? (I’m not using >>>> PRAgMaTIc to do mesh adaptation, so I was hoping to not run into an error) >>>> >>> >>> I don't think its broken. >>> >>> >>> Oh, OK. Could you help me figure out what’s the problem then, e.g., with >>> a slight (command line) variation of test #6, please? >>> >> >> Sure. I am at SIAM this week, but as soon as I can I will get you the fix. >> >> >>> $ cd src/dm/impls/plex/examples/tests >>> $ git diff ex19.c >>> $ make ex19 >>> $ mpirun ./ex19 -dim 3 -nbrVerEdge 10 -dm_plex_separate_marker 0 -met 0 >>> -hmin 0.1 -hmax 0.3 -init_dm_view -adapt_dm_view -do_L2 -petscspace_degree >>> 1 -petscfe_default_quadrature_order 1 -dm_plex_hash_location >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: The section point closure size 0 != dual space dimension >>> 4 >>> […] >>> [0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 633 in >>> src/dm/impls/plex/plexproject.c >>> [0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 771 in >>> src/dm/impls/plex/plexproject.c >>> [0]PETSC ERROR: #3 DMProjectFunctionLocal() line 7809 in >>> src/dm/interface/dm.c >>> [0]PETSC ERROR: #4 DMProjectFunction() line 7766 in src/dm/interface/dm.c >>> >>> If I comment the DMProjectFunction() call, I end up with the same error >>> as in my first message in DMCreateInterpolation(). >>> >>> 2) is DMCreateInterpolation + MatInterpolate the correct way of >>>> transferring one Vec from a DMPlex onto another? >>>> >>> >>> That is the intent. >>> >>> >>>> 3) if yes, by looking at the names of the arguments in >>>> DMPlexComputeInterpolatorGeneral, dmc and dmf, could you comment on the >>>> performance of this function for unrelated meshes, e.g., if both DMs are >>>> “fine” and not one coarse and the other fine (albeit non-nested), for >>>> simple P_k spaces. >>>> >>> >>> In general, it is going to be horrible. Here is what it does: locate the >>> fine quadrature points in the coarse grid and interpolate to them. This >>> quadrature can have huge errors if it falls across multiple cells. This is >>> why the nested version works perfectly, and also why Patrick Farrell and >>> James Maddison have the Supermesh library, which makes a refinement of the >>> mesh until the quadrature is accurate everywhere. That way they guarantee >>> that at least the zeroth moment is preserved. >>> >>> >>> Two subquestions if I may: >>> 1) are there any plans to have this integrated through an external >>> package? >>> >> >> In the absence of a plan, there is a hope. I would really like it to >> happen. >> >> >>> 2) if I understand you correctly, you answered about the numerical >>> performance of the function. I can live with high interpolation errors if >>> both meshes are “far" from each other. I was mostly interested in the >>> parallel performance of the function. >>> >> >> Everything is purely local except for point location. Since it has never >> really been tested in this mode, I am sure the >> scaling can be bad. I believe the default is to extrapolate if the point >> is not covered, which makes sense for mostly >> matching meshes. There is parallel point location, but it is intended for >> a few points where we are sampling the solution, >> rather than lots of points everywhere which you would get for >> non-matching meshes with different distributions. Could >> you say what kind of situation you are trying to optimize for? >> > > Matt, probably clarifying the parallel point location algorithm is helpful. > > 1/ Does it broadcast all off rank points to every rank? > Or > 2/ Does it broadcast sub domain bounding boxes to every rank, and then > scatter points to candidate owning ranks based on the bounding boxes > containing off rank points? > > I recall the method (years ago) did what’s described in (1) > I thought it was 2. This is one of the things on the list when we scale up the plasmas physics code. My reading says that a hierarchy of regular divisions is just as efficient for location as a structure for irregular divisons, like a k-d tree, and we can keep the bottom part of the hierarchy everywhere. Is that your feeling? Matt > >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Pierre >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks in advance for your help, >>>> Pierre >>>> >>>> $ mpirun -n 1 ./ex19 -msh in.msh -init_dm_view ::ascii_info >>>> -adapt_dm_view ::ascii_info -mat_view ::ascii_info -do_L2 >>>> -petscspace_degree 1 >>>> DM Object: DMinit 1 MPI processes >>>> type: plex >>>> DMinit in 3 dimensions: >>>> 0-cells: 1331 >>>> 1-cells: 7930 >>>> 2-cells: 12600 >>>> 3-cells: 6000 >>>> Labels: >>>> depth: 4 strata with value/size (0 (1331), 1 (7930), 2 (12600), 3 >>>> (6000)) >>>> Face Sets: 6 strata with value/size (4 (200), 1 (200), 5 (200), 2 >>>> (200), 3 (200), 6 (200)) >>>> Cell Sets: 1 strata with value/size (0 (6000)) >>>> DM Object: DMadapt (adapt_) 1 MPI processes >>>> type: plex >>>> DMadapt in 3 dimensions: >>>> 0-cells: 2905 >>>> 1-cells: 18888 >>>> 2-cells: 31368 >>>> 3-cells: 15384 >>>> Labels: >>>> depth: 4 strata with value/size (0 (2905), 1 (18888), 2 (31368), 3 >>>> (15384)) >>>> Face Sets: 6 strata with value/size (1 (200), 4 (200), 6 (200), 2 >>>> (200), 5 (200), 3 (200)) >>>> Cell Sets: 1 strata with value/size (0 (15384)) >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
