Dear Matt,
I have been working on this now with Petsc-3.18.3
1) I can confirm that enforcing periodicity works for a single core
simulation.
2) However, when using multiple cores, the code still hangs. Is there
something I should do to fix this? Or should this be fixed in the next
Petsc version?
3) This is strange, as it works fine for me.
Thanks, best, Berend.
On 12/15/22 18:56, Matthew Knepley wrote:
On Wed, Dec 14, 2022 at 3:58 AM Berend van Wachem
<[email protected] <mailto:[email protected]>> wrote:
Dear PETSc team and users,
I have asked a few times about this before, but we haven't really
gotten
this to work yet.
In our code, we use the DMPlex framework and are also interested in
periodic geometries.
As our simulations typically require many time-steps, we would like to
be able to save the DM to file and to read it again to resume the
simulation (a restart).
Although this works for a non-periodic DM, we haven't been able to get
this to work for a periodic one. To illustrate this, I have made a
working example, consisting of 2 files, createandwrite.c and
readandcreate.c. I have attached these 2 working examples. We are using
Petsc-3.18.2.
In the first file (createandwrite.c) a DMPlex is created and written to
a file. Periodicity is activated on lines 52-55 of the code.
In the second file (readandcreate.c) a DMPlex is read from the file.
When a periodic DM is read, this does not work. Also, trying to
'enforce' periodicity, lines 55 - 66, does not work if the number of
processes is larger than 1 - the code "hangs" without producing an
error.
Could you indicate what I am missing? I have really tried many
different
options, without finding a solution.
Hi Berend,
There are several problems. I will eventually fix all of them, but I
think we can get this working quickly.
1) Periodicity information is not saved. I will fix this, but forcing it
should work.
2) You were getting a hang because the blocksize on the local
coordinates was not set correctly after loading
since the vector had zero length. This does not happen in any test
because HDF5 loads a global vector, but
most other things create local coordinates. I have a fix for this,
which I will get in an MR, Also, I moved DMLocalizeCoordinates()
after distribution, since this is where it belongs.
knepley/fix-plex-periodic-faces *$:/PETSc3/petsc/petsc-dev$ git diff
diff --git a/src/dm/interface/dmcoordinates.c
b/src/dm/interface/dmcoordinates.c
index a922348f95b..6437e9f7259 100644
--- a/src/dm/interface/dmcoordinates.c
+++ b/src/dm/interface/dmcoordinates.c
@@ -551,10 +551,14 @@ PetscErrorCode DMGetCoordinatesLocalSetUp(DM dm)
PetscFunctionBegin;
PetscValidHeaderSpecific(dm, DM_CLASSID, 1);
if (!dm->coordinates[0].xl && dm->coordinates[0].x) {
- DM cdm = NULL;
+ DM cdm = NULL;
+ PetscInt bs;
PetscCall(DMGetCoordinateDM(dm, &cdm));
PetscCall(DMCreateLocalVector(cdm, &dm->coordinates[0].xl));
+ // If the size of the vector is 0, it will not get the right block size
+ PetscCall(VecGetBlockSize(dm->coordinates[0].x, &bs));
+ PetscCall(VecSetBlockSize(dm->coordinates[0].xl, bs));
PetscCall(PetscObjectSetName((PetscObject)dm->coordinates[0].xl,
"coordinates"));
PetscCall(DMGlobalToLocalBegin(cdm, dm->coordinates[0].x,
INSERT_VALUES, dm->coordinates[0].xl));
PetscCall(DMGlobalToLocalEnd(cdm, dm->coordinates[0].x,
INSERT_VALUES, dm->coordinates[0].xl));
3) If I comment out forcing the periodicity, your example does not run
for me. I will try to figure it out
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: SF roots 4400 < pEnd 6000
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: WARNING! There are option(s) set that were not used!
Could be the program crashed before they were used or a spelling
mistake, etc!
[1]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Option left: name:-start_in_debugger_no (no value)
source: command line
[1]PETSC ERROR: SF roots 4400 < pEnd 6000
[0]PETSC ERROR: See https://petsc.org/release/faq/
<https://petsc.org/release/faq/> for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0
GIT Date: 2022-12-12 23:42:20 +0000
[1]PETSC ERROR: WARNING! There are option(s) set that were not used!
Could be the program crashed before they were used or a spelling
mistake, etc!
[1]PETSC ERROR: Option left: name:-start_in_debugger_no (no value)
source: command line
[0]PETSC ERROR: ./readandcreate on a arch-master-debug named
MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by knepley
Thu Dec 15 12:50:26 2022
[1]PETSC ERROR: See https://petsc.org/release/faq/
<https://petsc.org/release/faq/> for trouble shooting.
[0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug
--download-bamg --download-bison --download-chaco --download-ctetgen
--download-egads --download-eigen --download-exodusii --download-fftw
--download-hpddm --download-ks --download-libceed --download-libpng
--download-metis --download-ml --download-mumps --download-muparser
--download-netcdf --download-opencascade --download-p4est
--download-parmetis --download-pnetcdf --download-pragmatic
--download-ptscotch --download-scalapack --download-slepc
--download-suitesparse --download-superlu_dist --download-tetgen
--download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake
--with-ctest-exec=/PETSc3/petsc/apple/bin/ctest
--with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple
--with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib
[1]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0
GIT Date: 2022-12-12 23:42:20 +0000
[0]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at
/PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322
[1]PETSC ERROR: ./readandcreate on a arch-master-debug named
MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by knepley
Thu Dec 15 12:50:26 2022
[0]PETSC ERROR: #2 DMGetGlobalSection() at
/PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527
[1]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug
--download-bamg --download-bison --download-chaco --download-ctetgen
--download-egads --download-eigen --download-exodusii --download-fftw
--download-hpddm --download-ks --download-libceed --download-libpng
--download-metis --download-ml --download-mumps --download-muparser
--download-netcdf --download-opencascade --download-p4est
--download-parmetis --download-pnetcdf --download-pragmatic
--download-ptscotch --download-scalapack --download-slepc
--download-suitesparse --download-superlu_dist --download-tetgen
--download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake
--with-ctest-exec=/PETSc3/petsc/apple/bin/ctest
--with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple
--with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib
[0]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750
[1]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at
/PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322
[0]PETSC ERROR: #4 DMPlexSectionLoad() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364
[1]PETSC ERROR: #2 DMGetGlobalSection() at
/PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527
[0]PETSC ERROR: #5 main() at
/Users/knepley/Downloads/tmp/Berend/readandcreate.c:85
[1]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -malloc_debug (source: environment)
[1]PETSC ERROR: #4 DMPlexSectionLoad() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364
[1]PETSC ERROR: #5 main() at
/Users/knepley/Downloads/tmp/Berend/readandcreate.c:85
[0]PETSC ERROR: -start_in_debugger_no (source: command line)
[1]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to [email protected]
[1]PETSC ERROR: -malloc_debug (source: environment)
application called MPI_Abort(MPI_COMM_SELF, 60) - process 0
[1]PETSC ERROR: -start_in_debugger_no (source: command line)
[1]PETSC ERROR: ----------------End of Error Message -------send entire
error message to [email protected]
application called MPI_Abort(MPI_COMM_SELF, 60) - process 0
4) We now have parallel HDF5 loading, so you should not have to manually
distribute. I will change your example to use it
and send it back when I am done.
Thanks!
Matt
Many thanks and kind regards,
Berend.
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
