Hi Matt, I’m running into error while using DMPlexNaturalToGlobalBegin/End and am hoping you have some insights in what I’m doing incorrectly. I create a 2x2x2 grid and distribute it across processors (N=1,2). I create a natural and a global vector; and then call DMPlexNaturalToGlobalBegin/End. Here are the two issues:
- When N = 1, PETSc complains about DMSetUseNatural() not being called before DMPlexDistribute(), which is certainly not the case. - For N=1 and 2, global vector doesn’t have valid entries. I’m not sure how to create the natural vector and have used DMCreateGlobalVector() to create the natural vector, which could be the issue. Attached is the sample code to reproduce the error and below is the screen output. >make ex_test ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 1 ./ex_test Natural vector: Vec Object: 1 MPI processes type: seq 0. 1. 2. 3. 4. 5. 6. 7. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: DM global to natural SF was not created. You must call DMSetUseNatural() before DMPlexDistribute(). [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.12.2-537-g5f77d1e0e5 GIT Date: 2019-12-21 14:33:27 -0600 [0]PETSC ERROR: ./ex_test on a darwin-gcc8 named WE37411 by bish218 Wed Jan 15 12:34:03 2020 [0]PETSC ERROR: Configure options --with-blaslapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-hdf5-dir=/opt/local --download-zlib --download-exodusii=yes --download-hdf5=yes --download-netcdf=yes --download-pnetcdf=yes --download-hypre=yes --download-mpich=yes --download-mumps=yes --download-scalapack=yes --with-cc=/opt/local/bin/gcc-mp-8 --with-cxx=/opt/local/bin/g++-mp-8 --with-fc=/opt/local/bin/gfortran-mp-8 --download-sowing=1 PETSC_ARCH=darwin-gcc8 [0]PETSC ERROR: #1 DMPlexNaturalToGlobalBegin() line 289 in /Users/bish218/projects/petsc/petsc_v3.12.2/src/dm/impls/plex/plexnatural.c Global vector: Vec Object: 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 0. 0. Information about the mesh: Rank = 0 local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1 local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1 local_id = 02; (0.250000, 0.750000, 0.250000); is_local = 1 local_id = 03; (0.750000, 0.750000, 0.250000); is_local = 1 local_id = 04; (0.250000, 0.250000, 0.750000); is_local = 1 local_id = 05; (0.750000, 0.250000, 0.750000); is_local = 1 local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 1 local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 2 ./ex_test Natural vector: Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. 3. Process [1] 4. 5. 6. 7. Global vector: Vec Object: 2 MPI processes type: mpi Process [0] 0. 0. 0. 0. Process [1] 0. 0. 0. 0. Information about the mesh: Rank = 0 local_id = 00; (0.250000, 0.750000, 0.250000); is_local = 1 local_id = 01; (0.750000, 0.750000, 0.250000); is_local = 1 local_id = 02; (0.250000, 0.750000, 0.750000); is_local = 1 local_id = 03; (0.750000, 0.750000, 0.750000); is_local = 1 local_id = 04; (0.250000, 0.250000, 0.250000); is_local = 0 local_id = 05; (0.750000, 0.250000, 0.250000); is_local = 0 local_id = 06; (0.250000, 0.250000, 0.750000); is_local = 0 local_id = 07; (0.750000, 0.250000, 0.750000); is_local = 0 Rank = 1 local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1 local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1 local_id = 02; (0.250000, 0.250000, 0.750000); is_local = 1 local_id = 03; (0.750000, 0.250000, 0.750000); is_local = 1 local_id = 04; (0.250000, 0.750000, 0.250000); is_local = 0 local_id = 05; (0.750000, 0.750000, 0.250000); is_local = 0 local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 0 local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -Gautam On Jan 9, 2020, at 4:57 PM, 'Bisht, Gautam' via tdycores-dev <[email protected]<mailto:[email protected]>> wrote: On Jan 9, 2020, at 4:25 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev <[email protected]<mailto:[email protected]>> wrote: > On Jan 9, 2020, at 2:58 PM, Jed Brown > <[email protected]<mailto:[email protected]>> wrote: > > "'Bisht, Gautam' via tdycores-dev" > <[email protected]<mailto:[email protected]>> writes: > >>> Do you need to rely on the element number, or would coordinates (of a >>> centroid?) be sufficient for your purposes? >> >> I do need to rely on the element number. In my case, I have a mapping file >> that remaps data from one grid onto another grid. Though I’m currently >> creating a hexahedron mesh, in the future I would be reading in an >> unstructured grid from a file for which I cannot rely on coordinates. > > How does the mapping file work and how is it generated? In CESM/E3SM, the mapping file is used to map fluxes or states between grids of two components (e.g. land & atmosphere). The mapping method can be conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB script to create the mapping file. I’m surprised that this is not an issue for other codes that are using DMPlex. E.g In PFLOTRAN, when a user creates a custom unstructured grid, they can specify material property for each grid cell. So, there should be a way to create a vectorscatter that will scatter material property read in the “application”-order (i.e. order before calling DMPlexDistribute() ) to ghosted-order (i.e. order after calling DMPlexDistribute()). We did build something specific for this because some people wanted it. I wish I could purge this from all simulations. Its definitely destructive, but this is the way the world currently is. You want this: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html<https://protect2.fireeye.com/v1/url?k=5fd6bd0b-036383c4-5fd6971e-0cc47adc5e60-2156f2075f6da02c&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fpetsc-current%2Fdocs%2Fmanualpages%2FDMPLEX%2FDMPlexNaturalToGlobalBegin.html> Perfect. Thanks. -Gautam Thanks, Matt > We can locate points and create interpolation with unstructured grids. > > -- > You received this message because you are subscribed to the Google Groups > "tdycores-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to > [email protected]<mailto:tdycores-dev%[email protected]>. > To view this discussion on the web visit > https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4&q=1&e=0962f8e1-9155-4d9c-abdf-2b6481141cd0&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org. -- You received this message because you are subscribed to the Google Groups "tdycores-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:tdycores-dev%[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov<https://protect2.fireeye.com/v1/url?k=711264f2-2da75a3d-71124ee7-0cc47adc5e60-a1731ca51d9b5203&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F9AB001AF-8857-446A-AE69-E8D6A25CB8FA%2540pnnl.gov>. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/> -- You received this message because you are subscribed to the Google Groups "tdycores-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gm%3DSY%3DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%2Bv-zVvFnA%40mail.gmail.com<https://protect2.fireeye.com/v1/url?k=ce43b754-92f6899b-ce439d41-0cc47adc5e60-0a864c0f80b4c75d&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2FCAMYG4Gm%253DSY%253DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%252Bv-zVvFnA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter>. -- You received this message because you are subscribed to the Google Groups "tdycores-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/7C23ABBA-2F76-4EAB-9834-9391AD77E18B%40pnnl.gov<https://protect2.fireeye.com/v1/url?k=365c6512-6ae95bdd-365c4f07-0cc47adc5e60-f9b3b6f5e8918ac5&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F7C23ABBA-2F76-4EAB-9834-9391AD77E18B%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter>.
ex_test.c
Description: ex_test.c
makefile
Description: makefile
