On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland < [email protected]> wrote:
> Hi Matthew, > > we tried to reproduce the error in a simple example. > > The context is the following: We hard coded the mesh and initial partition > into the code (see sConnectivity and sInitialPartition) for 2 ranks and try > to create a section in order to use the DMPlexNaturalToGlobalBegin function > to retreive our initial element numbers. > > Now the call to DMPlexDistribute give different errors depending on what > type of component we ask the field to be created. For our objective, we > would like a global field to be created on elements only (like a P0 > interpolation). > > We now have the following error generated: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Inconsistency in indices, 18 should be 17 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021 > [0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 14:52:36 2021 > [0]PETSC ERROR: Configure options > --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 --with-mpi-compilers=1 > --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 --with-cxx-dialect=C++14 > --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes > --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 > --download-ml=yes --download-mumps=yes --download-superlu=yes > --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes > --download-parmetis=yes --download-ptscotch=yes --download-metis=yes > --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes > --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 > --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. > --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. > --with-scalapack=1 > --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include > --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 > -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" > [0]PETSC ERROR: #1 PetscSFCreateSectionSF() at > /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 > [0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at > /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 > [0]PETSC ERROR: #3 DMPlexDistribute() at > /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 > [0]PETSC ERROR: #4 main() at bug_section.cc:159 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to [email protected] > > Hope the attached code is self-explaining, note that to make it short, we > have not included the final part of it, just the buggy part we are > encountering right now... > > Thanks for your insights, > > Thanks for making the example. I tweaked it slightly. I put in a test case that just makes a parallel 7 x 10 quad mesh. This works fine. Thus I think it must be something connected with the original mesh. It is hard to get a handle on it without the coordinates. Do you think you could put the coordinate array in? I have added the code to load them (see attached file). Thanks, Matt > Eric > On 2021-10-06 9:23 p.m., Matthew Knepley wrote: > > On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland < > [email protected]> wrote: > >> Hi Matthew, >> >> we tried to use that. Now, we discovered that: >> >> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, it is >> not created because DMPlexCreateGlobalToNaturalSF looks for a "section": >> this is not documented in DMSetUseNaturalso we are asking ourselfs: "is >> this a permanent feature or a temporary situation?" >> > I think explaining this will help clear up a lot. > > What the Natural2Global map does is permute a solution vector into the > ordering that it would have had prior to mesh distribution. > Now, in order to do this permutation, I need to know the original (global) > data layout. If it is not specified _before_ distribution, we > cannot build the permutation. The section describes the data layout, so I > need it before distribution. > > I cannot think of another way that you would implement this, but if you > want something else, let me know. > >> 2- We then tried to create a "section" in different manners: we took the >> code into the example petsc/src/dm/impls/plex/tests/ex15.c. However, we >> ended up with a segfault: >> >> corrupted size vs. prev_size >> [rohan:07297] *** Process received signal *** >> [rohan:07297] Signal: Aborted (6) >> [rohan:07297] Signal code: (-6) >> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >> [rohan:07297] [ 8] /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >> [rohan:07297] [ 9] >> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >> [rohan:07297] [10] >> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >> [rohan:07297] [11] >> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >> [rohan:07297] [12] >> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >> >> [rohan:07297] [14] >> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >> [rohan:07297] [15] >> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >> [rohan:07297] [16] >> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >> [rohan:07297] [17] >> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >> [rohan:07297] [18] >> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >> > I am not sure what happened here, but if you could send a sample code, I > will figure it out. > >> If we do not create a section, the call to DMPlexDistribute is >> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer... >> > Yes, it just ignores it in this case because it does not have a global > layout. > >> Here are the operations we are calling ( this is almost the code we are >> using, I just removed verifications and creation of the connectivity which >> use our parallel structure and code): >> >> =========== >> >> PetscInt* lCells = 0; >> PetscInt lNumCorners = 0; >> PetscInt lDimMail = 0; >> PetscInt lnumCells = 0; >> >> //At this point we create the cells for PETSc expected input for >> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells >> to correct values. >> ... >> >> DM lDMBete = 0 >> DMPlexCreate(lMPIComm,&lDMBete); >> >> DMSetDimension(lDMBete, lDimMail); >> >> DMPlexBuildFromCellListParallel(lDMBete, >> lnumCells, >> PETSC_DECIDE, >> >> pLectureElementsLocaux.reqNbTotalSommets(), >> lNumCorners, >> lCells, >> PETSC_NULL); >> >> DM lDMBeteInterp = 0; >> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >> DMDestroy(&lDMBete); >> lDMBete = lDMBeteInterp; >> >> DMSetUseNatural(lDMBete,PETSC_TRUE); >> >> PetscSF lSFMigrationSansOvl = 0; >> PetscSF lSFMigrationOvl = 0; >> DM lDMDistribueSansOvl = 0; >> DM lDMAvecOverlap = 0; >> >> PetscPartitioner lPart; >> DMPlexGetPartitioner(lDMBete, &lPart); >> PetscPartitionerSetFromOptions(lPart); >> >> PetscSection section; >> PetscInt numFields = 1; >> PetscInt numBC = 0; >> PetscInt numComp[1] = {1}; >> PetscInt numDof[4] = {1, 0, 0, 0}; >> PetscInt bcFields[1] = {0}; >> IS bcPoints[1] = {NULL}; >> >> DMSetNumFields(lDMBete, numFields); >> >> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, bcFields, >> bcPoints, NULL, NULL, §ion); >> DMSetLocalSection(lDMBete, section); >> >> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl, >> &lDMDistribueSansOvl); // segfault! >> >> =========== >> >> So we have other question/remarks: >> >> 3- Maybe PETSc expect something specific that is missing/not verified: >> for example, we didn't gave any coordinates since we just want to partition >> and compute overlap for the mesh... and then recover our element numbers in >> a "simple way" >> >> 4- We are telling ourselves it is somewhat a "big price to pay" to have >> to build an unused section to have the global to natural ordering set ? >> Could this requirement be avoided? >> > I don't think so. There would have to be _some_ way of describing your > data layout in terms of mesh points, and I do not see how you could use > less memory doing that. > >> 5- Are there any improvement towards our usages in 3.16 release? >> > Let me try and run the code above. > > Thanks, > > Matt > >> Thanks, >> >> Eric >> >> >> On 2021-09-29 7:39 p.m., Matthew Knepley wrote: >> >> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland < >> [email protected]> wrote: >> >>> Hi, >>> >>> I come back with _almost_ the original question: >>> >>> I would like to add an integer information (*our* original element >>> number, not petsc one) on each element of the DMPlex I create with >>> DMPlexBuildFromCellListParallel. >>> >>> I would like this interger to be distribruted by or the same way >>> DMPlexDistribute distribute the mesh. >>> >>> Is it possible to do this? >>> >> >> I think we already have support for what you want. If you call >> >> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >> >> before DMPlexDistribute(), it will compute a PetscSF encoding the global >> to natural map. You >> can get it with >> >> >> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >> >> and use it with >> >> >> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >> >> Is this sufficient? >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Eric >>> >>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote: >>> > Hi, >>> > >>> > I want to use DMPlexDistribute from PETSc for computing overlapping >>> > and play with the different partitioners supported. >>> > >>> > However, after calling DMPlexDistribute, I noticed the elements are >>> > renumbered and then the original number is lost. >>> > >>> > What would be the best way to keep track of the element renumbering? >>> > >>> > a) Adding an optional parameter to let the user retrieve a vector or >>> > "IS" giving the old number? >>> > >>> > b) Adding a DMLabel (seems a wrong good solution) >>> > >>> > c) Other idea? >>> > >>> > Of course, I don't want to loose performances with the need of this >>> > "mapping"... >>> > >>> > Thanks, >>> > >>> > Eric >>> > >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
ex44.c
Description: Binary data
