On Wed, Nov 24, 2021 at 5:45 PM Bhargav Subramanya < [email protected]> wrote:
> On Wed, Nov 24, 2021 at 8:59 PM Matthew Knepley <[email protected]> wrote: > >> On Wed, Nov 24, 2021 at 12:27 PM Bhargav Subramanya < >> [email protected]> wrote: >> >>> Dear Matt and Mark, >>> >>> Thank you very much for your reply. Your inputs are very useful to me. >>> >>> On Mon, Nov 22, 2021 at 9:38 PM Matthew Knepley <[email protected]> >>> wrote: >>> >>>> On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < >>>> [email protected]> wrote: >>>> >>>>> Dear All, >>>>> >>>>> I have a prismatic mesh generated by extruding a base icosahedron >>>>> spherical surface mesh. The mesh is distributed on 2 processes after >>>>> extrusion. The dof for my problem are defined only on the simplices of the >>>>> spherical layers. >>>>> >>>>> 1. For a given spherical layer, I want to get points, which are >>>>> basically simplices, lying on that layer only. DMPlexGetHeightStratum >>>>> returns the points on all the spherical Layers. I can probably use >>>>> DMPlexFindVertices (based on the radius of the spherical layer) and >>>>> DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to >>>>> retrieve the points (which are simplices) on a given spherical layer in >>>>> the >>>>> extruded mesh? >>>>> >>>> >>>> DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which >>>> is a DAG, not in the mesh. For example, height 0 points are the cells, >>>> height 1 are the faces, etc. >>>> >>>> I believe the default numbering for extrusion (in the main branch), is >>>> that all vertices produced from a given vertex be in order. This would mean >>>> that if v were the vertex point number, then >>>> >>>> (v - vStart) % Nl == l >>>> >>>> where Nl is the number of layers and l is the layer of that vertex. It >>>> is also the same for triangles. So if you want to segregate each shell, >>>> after extrusion, make a label that gives triangles this marker, and then >>>> use DMPlexLabelComplete(). Then after refinement you will still have the >>>> shells labeled correctly. >>>> >>>> I would be happy to help you make an example that does this. It seems >>>> cool. >>>> >>> >>> ----- Sure, I am interested to make an example of it. And, I really >>> appreciate your help. >>> >>>> >>>> >>>>> 2. I am able to visualize the entire mesh using dm_view. Is there a >>>>> way to get the mesh file for the local dm from a specific process? >>>>> >>>> >>>> You can use -dm_partition_view which outputs a field with the process >>>> number. Then use Clip inside Paraview and clip to the process number you >>>> want, >>>> or just view that field so each process has a different color. >>>> >>>> >>>>> 3. One of my previous emails might have got lost as I mistakenly >>>>> attached a large mesh file and sent it. So I am repeating the question >>>>> here. DMPlexExtrude gives the following error after distributing the base >>>>> 2D spherical mesh. Both refinement or/and extrusion after dm distribution >>>>> does not work for me. In fact, I tried >>>>> with src/dm/impls/plex/tutorials/ex10.c also. Although the mesh is >>>>> distributed after the extrusion in ex10.c (which is working fine for me), >>>>> I >>>>> tried to distribute before extrusion, and I get the following error. Could >>>>> you please suggest where I might be making a mistake? >>>>> >>>> >>>> So you want to distribute the mesh before extruding. For that small >>>> example (using the main branch), I run >>>> >>>> mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 >>>> -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 >>>> >>> >>> ----- I pulled the code from the main branch and configured my petsc >>> again. I am now finally able to reproduce the mesh that you have attached. >>> I am able to do parallel mesh refinement, followed by extrusion, and use >>> the following code to check that. However, I still have one problem. >>> Attached are the extruded mesh files. >>> >>> 1. Mesh doesn't seem to be refined properly near interfaces between the >>> processes. I then used -init_dm_distribute_overlap of 1 and that fixed the >>> problem. I hope this is the way to do that. >>> >> >> Hmm, something is wrong here. We should track this down and fix it. I do >> not see this. Could you tell me how to reproduce the problem? >> > > ----- I mainly wanted to check if the mesh was distributed or not before > the extrusion. I do this by specifying different prefix options as shown in > the code below. I think using different prefix options is probably causing > the problem. > > #include <petsc.h> > > int main(int argc, char **argv) > { > DM dm; > PetscBool distributed; > PetscErrorCode ierr; > > > ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; > ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); > ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); > ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, > "init_");CHKERRQ(ierr); > ierr = DMSetFromOptions(dm);CHKERRQ(ierr); > > ierr = DMPlexIsDistributed(dm, &distributed); > ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); > CHKERRQ(ierr); > ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); > > ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); > ierr = DMSetFromOptions(dm);CHKERRQ(ierr); > ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); > ierr = DMDestroy(&dm);CHKERRQ(ierr); > ierr = PetscFinalize(); > return ierr; > } > > The command line options are: mpiexec -n 3 ./cavity_flow.out > -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 > -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_extrude 3 > -dm_partition_view -dm_view hdf5:mesh.h5 > Okay, I run exactly this code and command on 'main', and it does work for me: master *:~/Downloads/tmp/hpc1$ /PETSc3/petsc/apple/bin/mpiexec -n 3 ./meshtest -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -init_dm_view -dm_refine 2 -dm_extrude 3 -dm_partition_view_no -dm_view -malloc_debug 0 DM Object: sphere (init_) 3 MPI processes type: plex sphere in 2 dimensions: 0-cells: 66 66 67 1-cells: 171 173 172 2-cells: 106 108 106 Labels: depth: 3 strata with value/size (0 (66), 1 (171), 2 (106)) celltype: 3 strata with value/size (0 (66), 1 (171), 3 (106)) 1 1 1 DM Object: sphere 3 MPI processes type: plex sphere in 3 dimensions: 0-cells: 3588 3636 3604 1-cells: 13059 (2691) 13271 (2727) 13087 (2703) 2-cells: 14560 (7776) 14820 (7908) 14572 (7788) 3-cells: 5088 5184 5088 Labels: celltype: 6 strata with value/size (5 (7776), 9 (5088), 0 (3588), 1 (10368), 3 (6784), 2 (2691)) depth: 4 strata with value/size (0 (3588), 1 (13059), 2 (14560), 3 (5088)) WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-dm_partition_view_no (no value) > 2. The parallel mesh refinement doesn't seem to take care of the spherical >>> geometry (shown in the attached mesh file). Could you please suggest how to >>> fix it? >>> >> >> Okay, this is more subtle. Let me explain how things work inside. >> >> 1) When you refine, it inserts new cells/faces/edges/vertices. When we >> insert a new vertex, we ask what the coordinates should be. If no special >> information is there, we >> just average the surrounding vertices. >> >> 2) Then we allow a second step that processed all coordinates in the mesh >> after refinement >> >> For the sphere, we use 2) to make all points stick to the implicit >> surface (radius R). We cannot do this for the extruded mesh, so we turn it >> off. >> > > ---- thanks for this explanation. > > >> What makes the most sense to me is to do >> > >> 1) Serial mesh refinement to get enough cells to distribute evenly >> >> 2) Mesh distribution >> >> 3) Mesh refinement >> >> 4) Extrusion >> >> This should preserve the geometry and scale well. Does that sound right? >> > > -- Yes, this sounds right, and this must preserve the geometry. In fact, I > was under the assumption that I was doing the exact same thing by using > these options: > mpiexec -n 3 ./cavity_flow.out -init_dm_plex_shape sphere > -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute > -dm_refine 2 -dm_extrude 3 -dm_partition_view -dm_view hdf5:mesh.h5 > Do you think these options are not doing the things in the order of 1,2,3 > and 4 as you specified? If not, could you suggest the options that I need > to use to get in the order of 1,2,3 and 4? > Yes, I am getting the surface of the sphere with this. Thanks, Matt > >> Thanks, >> >> Matt >> >> >>> command line options: >>> -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 >>> -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_partition_view >>> -dm_view hdf5:mesh.h5 -dm_extrude 3 >>> >>> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >>> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >>> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, >>> "init_");CHKERRQ(ierr); >>> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >>> >>> ierr = DMPlexIsDistributed(dm, &distributed); >>> ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); >>> CHKERRQ(ierr); >>> ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); >>> >>> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); >>> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >>> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >>> >>> >>>> and I get the attached picture. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no >>>>> graph set >>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ >>>>> <https://urldefense.com/v3/__https://petsc.org/release/faq/__;!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOW2dk_COg$> >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown >>>>> [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named >>>>> kl-21859 by subrambm Mon Nov 22 19:47:14 2021 >>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>>>> --with-fc=gfortran --download-mpich --download-fblaslapack >>>>> --download-superlu_dist --download-hypre --download-fiat >>>>> --download-generator --download-triangle --download-tetgen >>>>> --download-chaco >>>>> --download-make -download-boost --download-cmake --download-ml >>>>> --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git >>>>> <https://urldefense.com/v3/__https://bitbucket.org/petsc/pkg-mumps.git__;!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOW414fjd0$> >>>>> --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch >>>>> --download-hdf5 --force >>>>> [0]PETSC ERROR: #1 DMPlexCheckPointSF() at >>>>> /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 >>>>> [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at >>>>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 >>>>> [0]PETSC ERROR: #3 DMPlexInterpolate() at >>>>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 >>>>> [0]PETSC ERROR: #4 DMPlexExtrude() at >>>>> /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 >>>>> [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 >>>>> [0]PETSC ERROR: #6 main() at ex10.c:180 >>>>> [0]PETSC ERROR: PETSc Option Table entries: >>>>> [0]PETSC ERROR: -dm_plex_extrude_layers 3 >>>>> [0]PETSC ERROR: -dm_view vtk:mesh.vtk >>>>> [0]PETSC ERROR: -init_dm_plex_dim 2 >>>>> [0]PETSC ERROR: -petscpartitioner_type simple >>>>> [0]PETSC ERROR: -srf_dm_distribute >>>>> [0]PETSC ERROR: -srf_dm_refine 0 >>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>> entire error message to [email protected] >>>>> >>>>> Thanks, >>>>> Bhargav >>>>> >>>>> >>>>> ------------------------------ >>>>> This message and its contents, including attachments are intended >>>>> solely for the original recipient. If you are not the intended recipient >>>>> or >>>>> have received this message in error, please notify me immediately and >>>>> delete this message from your computer system. Any unauthorized use or >>>>> distribution is prohibited. Please consider the environment before >>>>> printing >>>>> this email. >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOWdEg9at8$> >>>> >>> >>> ------------------------------ >>> This message and its contents, including attachments are intended solely >>> for the original recipient. If you are not the intended recipient or have >>> received this message in error, please notify me immediately and delete >>> this message from your computer system. Any unauthorized use or >>> distribution is prohibited. Please consider the environment before printing >>> this email. >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!Nmw4Hv0!jtcvFDJFDmYH37N1wm6apnWSZeA9Tu6jsXTTbAqZ7w6gvNxKzNSfHoeVDIUfLDjsDO1vgciyMAw$> >> > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
