On Tue, Jan 24, 2023 at 10:39 AM Berend van Wachem <[email protected]> wrote:
> Dear Matt, > > I have been working on this now with Petsc-3.18.3 > > 1) I can confirm that enforcing periodicity works for a single core > simulation. > > 2) However, when using multiple cores, the code still hangs. Is there > something I should do to fix this? Or should this be fixed in the next > Petsc version? > Dang dang dang. I forgot to merge this fix. Thanks for reminding me. It is now here: https://gitlab.com/petsc/petsc/-/merge_requests/6001 > 3) This is strange, as it works fine for me. > Will try again with current main. Thanks Matt > Thanks, best, Berend. > > > On 12/15/22 18:56, Matthew Knepley wrote: > > On Wed, Dec 14, 2022 at 3:58 AM Berend van Wachem > > <[email protected] <mailto:[email protected]>> wrote: > > > > > > Dear PETSc team and users, > > > > I have asked a few times about this before, but we haven't really > > gotten > > this to work yet. > > > > In our code, we use the DMPlex framework and are also interested in > > periodic geometries. > > > > As our simulations typically require many time-steps, we would like > to > > be able to save the DM to file and to read it again to resume the > > simulation (a restart). > > > > Although this works for a non-periodic DM, we haven't been able to > get > > this to work for a periodic one. To illustrate this, I have made a > > working example, consisting of 2 files, createandwrite.c and > > readandcreate.c. I have attached these 2 working examples. We are > using > > Petsc-3.18.2. > > > > In the first file (createandwrite.c) a DMPlex is created and written > to > > a file. Periodicity is activated on lines 52-55 of the code. > > > > In the second file (readandcreate.c) a DMPlex is read from the file. > > When a periodic DM is read, this does not work. Also, trying to > > 'enforce' periodicity, lines 55 - 66, does not work if the number of > > processes is larger than 1 - the code "hangs" without producing an > > error. > > > > Could you indicate what I am missing? I have really tried many > > different > > options, without finding a solution. > > > > > > Hi Berend, > > > > There are several problems. I will eventually fix all of them, but I > > think we can get this working quickly. > > > > 1) Periodicity information is not saved. I will fix this, but forcing it > > should work. > > > > 2) You were getting a hang because the blocksize on the local > > coordinates was not set correctly after loading > > since the vector had zero length. This does not happen in any test > > because HDF5 loads a global vector, but > > most other things create local coordinates. I have a fix for this, > > which I will get in an MR, Also, I moved DMLocalizeCoordinates() > > after distribution, since this is where it belongs. > > > > knepley/fix-plex-periodic-faces *$:/PETSc3/petsc/petsc-dev$ git diff > > diff --git a/src/dm/interface/dmcoordinates.c > > b/src/dm/interface/dmcoordinates.c > > index a922348f95b..6437e9f7259 100644 > > --- a/src/dm/interface/dmcoordinates.c > > +++ b/src/dm/interface/dmcoordinates.c > > @@ -551,10 +551,14 @@ PetscErrorCode DMGetCoordinatesLocalSetUp(DM dm) > > PetscFunctionBegin; > > PetscValidHeaderSpecific(dm, DM_CLASSID, 1); > > if (!dm->coordinates[0].xl && dm->coordinates[0].x) { > > - DM cdm = NULL; > > + DM cdm = NULL; > > + PetscInt bs; > > > > PetscCall(DMGetCoordinateDM(dm, &cdm)); > > PetscCall(DMCreateLocalVector(cdm, &dm->coordinates[0].xl)); > > + // If the size of the vector is 0, it will not get the right block > size > > + PetscCall(VecGetBlockSize(dm->coordinates[0].x, &bs)); > > + PetscCall(VecSetBlockSize(dm->coordinates[0].xl, bs)); > > PetscCall(PetscObjectSetName((PetscObject)dm->coordinates[0].xl, > > "coordinates")); > > PetscCall(DMGlobalToLocalBegin(cdm, dm->coordinates[0].x, > > INSERT_VALUES, dm->coordinates[0].xl)); > > PetscCall(DMGlobalToLocalEnd(cdm, dm->coordinates[0].x, > > INSERT_VALUES, dm->coordinates[0].xl)); > > > > 3) If I comment out forcing the periodicity, your example does not run > > for me. I will try to figure it out > > > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Nonconforming object sizes > > [0]PETSC ERROR: SF roots 4400 < pEnd 6000 > > [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: WARNING! There are option(s) set that were not used! > > Could be the program crashed before they were used or a spelling > > mistake, etc! > > [1]PETSC ERROR: Nonconforming object sizes > > [0]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) > > source: command line > > [1]PETSC ERROR: SF roots 4400 < pEnd 6000 > > [0]PETSC ERROR: See https://petsc.org/release/faq/ > > <https://petsc.org/release/faq/> for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0 > > GIT Date: 2022-12-12 23:42:20 +0000 > > [1]PETSC ERROR: WARNING! There are option(s) set that were not used! > > Could be the program crashed before they were used or a spelling > > mistake, etc! > > [1]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) > > source: command line > > [0]PETSC ERROR: ./readandcreate on a arch-master-debug named > > MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by knepley > > Thu Dec 15 12:50:26 2022 > > [1]PETSC ERROR: See https://petsc.org/release/faq/ > > <https://petsc.org/release/faq/> for trouble shooting. > > [0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug > > --download-bamg --download-bison --download-chaco --download-ctetgen > > --download-egads --download-eigen --download-exodusii --download-fftw > > --download-hpddm --download-ks --download-libceed --download-libpng > > --download-metis --download-ml --download-mumps --download-muparser > > --download-netcdf --download-opencascade --download-p4est > > --download-parmetis --download-pnetcdf --download-pragmatic > > --download-ptscotch --download-scalapack --download-slepc > > --download-suitesparse --download-superlu_dist --download-tetgen > > --download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake > > --with-ctest-exec=/PETSc3/petsc/apple/bin/ctest > > --with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple > > --with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib > > [1]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0 > > GIT Date: 2022-12-12 23:42:20 +0000 > > [0]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at > > /PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322 > > [1]PETSC ERROR: ./readandcreate on a arch-master-debug named > > MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by knepley > > Thu Dec 15 12:50:26 2022 > > [0]PETSC ERROR: #2 DMGetGlobalSection() at > > /PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527 > > [1]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug > > --download-bamg --download-bison --download-chaco --download-ctetgen > > --download-egads --download-eigen --download-exodusii --download-fftw > > --download-hpddm --download-ks --download-libceed --download-libpng > > --download-metis --download-ml --download-mumps --download-muparser > > --download-netcdf --download-opencascade --download-p4est > > --download-parmetis --download-pnetcdf --download-pragmatic > > --download-ptscotch --download-scalapack --download-slepc > > --download-suitesparse --download-superlu_dist --download-tetgen > > --download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake > > --with-ctest-exec=/PETSc3/petsc/apple/bin/ctest > > --with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple > > --with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib > > [0]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at > > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750 > > [1]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at > > /PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322 > > [0]PETSC ERROR: #4 DMPlexSectionLoad() at > > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364 > > [1]PETSC ERROR: #2 DMGetGlobalSection() at > > /PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527 > > [0]PETSC ERROR: #5 main() at > > /Users/knepley/Downloads/tmp/Berend/readandcreate.c:85 > > [1]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at > > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -malloc_debug (source: environment) > > [1]PETSC ERROR: #4 DMPlexSectionLoad() at > > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364 > > [1]PETSC ERROR: #5 main() at > > /Users/knepley/Downloads/tmp/Berend/readandcreate.c:85 > > [0]PETSC ERROR: -start_in_debugger_no (source: command line) > > [1]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to [email protected] > > [1]PETSC ERROR: -malloc_debug (source: environment) > > application called MPI_Abort(MPI_COMM_SELF, 60) - process 0 > > [1]PETSC ERROR: -start_in_debugger_no (source: command line) > > [1]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to [email protected] > > application called MPI_Abort(MPI_COMM_SELF, 60) - process 0 > > 4) We now have parallel HDF5 loading, so you should not have to manually > > distribute. I will change your example to use it > > and send it back when I am done. > > > > Thanks! > > > > Matt > > > > Many thanks and kind regards, > > Berend. > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ < > http://www.cse.buffalo.edu/~knepley/> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
