Re: [petsc-users] ex19: Segmentation Violation when run with MUMPS on MacOS (arm64)
The branch with the patched openmpi works fine for me. Thanks for sharing! Best wishes, Zongze > On 4 Apr 2024, at 12: 44, Satish Balay wrote: > > With xcode-15. 3 and branch "barry/2024-04-03/fix-chaco-modern-c/release" ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The branch with the patched openmpi works fine for me. Thanks for sharing! Best wishes, Zongze > On 4 Apr 2024, at 12:44, Satish Balay wrote: > > With xcode-15.3 and branch "barry/2024-04-03/fix-chaco-modern-c/release" from https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7433__;!!G_uCfscf7eWS!b2jrTotLzInMflHGijS61VKFqgwsNfFEui4cpKyEgYLspq5b8HyH38UJWxEuke5cGC4RnZf7jdYuh8CtFfZlodjP$ [and a patched openmpi tarball to remove -Wl,-commons,use_dylibs] the following works for me. > > Satish > > > > petsc@mpro petsc.x % ./configure --download-bison --download-chaco --download-ctetgen --download-eigen --download-fftw --download-hdf5 --download-hpddm --download-hwloc --download-hypre --download-libpng --download-metis --download-mmg --download-mumps --download-netcdf --download-openblas --download-openblas-make-options="'USE_THREAD=0 USE_LOCKING=1 USE_OPENMP=0'" --download-p4est --download-parmmg --download-pnetcdf --download-pragmatic --download-ptscotch --download-scalapack --download-slepc --download-suitesparse --download-superlu_dist --download-tetgen --download-triangle --with-c2html=0 --with-debugging=1 --with-fortran-bindings=0 --with-shared-libraries=1 --with-x=0 --with-zlib --download-openmpi=https://urldefense.us/v3/__https://web.cels.anl.gov/projects/petsc/download/externalpackages/openmpi-5.0.2-xcode15.tar.gz__;!!G_uCfscf7eWS!b2jrTotLzInMflHGijS61VKFqgwsNfFEui4cpKyEgYLspq5b8HyH38UJWxEuke5cGC4RnZf7jdYuh8CtFUMGGKvW$ --download-pastix && make && make check > > CC arch-darwin-c-debug/obj/src/lme/interface/lmesolve.o > CLINKER arch-darwin-c-debug/lib/libslepc.3.21.0.dylib >DSYMUTIL arch-darwin-c-debug/lib/libslepc.3.21.0.dylib > Now to install the library do: > make SLEPC_DIR=/Users/petsc/petsc.x/arch-darwin-c-debug/externalpackages/git.slepc PETSC_DIR=/Users/petsc/petsc.x install > = > *** Installing SLEPc *** > *** Installing SLEPc at prefix location: /Users/petsc/petsc.x/arch-darwin-c-debug *** > > Install complete. > Now to check if the libraries are working do (in current directory): > make SLEPC_DIR=/Users/petsc/petsc.x/arch-darwin-c-debug PETSC_DIR=/Users/petsc/petsc.x PETSC_ARCH=arch-darwin-c-debug check > > /usr/bin/make --no-print-directory -f makefile PETSC_ARCH=arch-darwin-c-debug PETSC_DIR=/Users/petsc/petsc.x SLEPC_DIR=/Users/petsc/petsc.x/arch-darwin-c-debug/externalpackages/git.slepc install-builtafterslepc > /usr/bin/make --no-print-directory -f makefile PETSC_ARCH=arch-darwin-c-debug PETSC_DIR=/Users/petsc/petsc.x SLEPC_DIR=/Users/petsc/petsc.x/arch-darwin-c-debug/externalpackages/git.slepc slepc4py-install > make[6]: Nothing to be done for `slepc4py-install'. > *** Building and installing HPDDM *** > = > Now to check if the libraries are working do: > make PETSC_DIR=/Users/petsc/petsc.x PETSC_ARCH=arch-darwin-c-debug check > = > Running PETSc check examples to verify correct installation > Using PETSC_DIR=/Users/petsc/petsc.x and PETSC_ARCH=arch-darwin-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE > C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS > C/C++ example src/snes/tutorials/ex19 run successfully with SuiteSparse > C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST > C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5 > Running SLEPc check examples to verify correct installation > Using SLEPC_DIR=/Users/petsc/petsc.x/arch-darwin-c-debug/externalpackages/git.slepc, PETSC_DIR=/Users/petsc/petsc.x, and PETSC_ARCH=arch-darwin-c-debug > C/C++ example src/eps/tests/test10 run successfully with 1 MPI process > C/C++ example src/eps/tests/test10 run successfully with 2 MPI processes > Completed SLEPc check examples > Completed PETSc check examples > petsc@mpro petsc.x % clang --version > Apple clang version 15.0.0 (clang-1500.3.9.4) > Target: arm64-apple-darwin23.4.0 > Thread model: posix > InstalledDir: /Library/Developer/CommandLineTools/usr/bin > petsc@mpro petsc.x % > &g
Re: [petsc-users] ex19: Segmentation Violation when run with MUMPS on MacOS (arm64)
Thank you for your update. I found some links that suggest this issue is related to the Apple linker, which is causing problems with Fortran linking. 1. https://urldefense.us/v3/__https://github.com/open-mpi/ompi/issues/12427__;!!G_uCfscf7eWS!bHY4uqpTfwl0jKopATQs3gw--TZSBmDp0Lb1gzDBeEu4ZB_zTph-jfw49yIr3jvPx0YEhQbk1PjYbGYVpjjms6Wb$ 2. https://urldefense.us/v3/__https://x.com/science_dot/status/1768667417553547635?s=46__;!!G_uCfscf7eWS!bHY4uqpTfwl0jKopATQs3gw--TZSBmDp0Lb1gzDBeEu4ZB_zTph-jfw49yIr3jvPx0YEhQbk1PjYbGYVptASYXS2$ Best wishes, Zongze > On 2 Apr 2024, at 01:15, Satish Balay wrote: > > On Mon, 1 Apr 2024, Zongze Yang wrote: > >> >> I noticed this in the config.log of OpenMPI: >> ``` >> configure:30230: checking to see if mpifort compiler needs additional linker >> flags >> configure:30247: gfortran -o conftest -fPIC -ffree-line-length-none >> -ffree-line-length-0 -Wno-lto-type-mismatch -g -O0 -fallow-argument-mismatch >> -Wl,-flat_namespace -Wl,-commons,use_dylibs conftest.f90 >&5 >> ld: warning: -commons use_dylibs is no longer supported, using error >> treatment instead >> configure:30247: $? = 0 >> configure:30299: result: -Wl,-commons,use_dylibs >> ``` >> So, I find it odd that this flag isn't picked up on your platform, as it >> only checked the exit value. > > I get: > > configure:30247: gfortran -o conftest -fPIC -ffree-line-length-none > -ffree-line-length-0 -Wno-lto-type-mismatch -g -O0 -fallow-argument-mismatch > -Wl,-flat_namespace -Wl,-commons,use_dylibs conftest.f90 >&5 > ld: unknown options: -commons > collect2: error: ld returned 1 exit status > configure:30247: $? = 1 > configure: failed program was: > | program test > | integer :: i > | end program > configure:30299: result: none > > Note, I have and older xcode-15/CLT version: > > petsc@npro ~ % clang --version > Apple clang version 15.0.0 (clang-1500.1.0.2.5) > Target: arm64-apple-darwin23.3.0 > Thread model: posix > InstalledDir: /Library/Developer/CommandLineTools/usr/bin > > Satish
Re: [petsc-users] ex19: Segmentation Violation when run with MUMPS on MacOS (arm64)
> On 1 Apr 2024, at 23: 38, Satish Balay wrote: > > On Sun, 31 Mar 2024, Zongze Yang wrote: >>> --- >>> petsc@ npro petsc % ./configure --download-bison --download-chaco --download-ctetgen ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd > On 1 Apr 2024, at 23:38, Satish Balay wrote: > > On Sun, 31 Mar 2024, Zongze Yang wrote: >>> --- >>> petsc@npro petsc % ./configure --download-bison --download-chaco --download-ctetgen --download-eigen --download-fftw --download-hdf5 --download-hpddm --download-hwloc --download-hypre --download-libpng --download-metis --download-mmg --download-mumps --download-netcdf --download-openblas >> --download-openblas-make-options="'USE_THREAD=0 USE_LOCKING=1 USE_OPENMP=0'" --download-p4est --download-parmmg --download-pnetcdf --download-pragmatic --download-ptscotch --download-scalapack --download-slepc --download-suitesparse --download-superlu_dist --download-tetgen --download-tri >> angle --with-c2html=0 --with-debugging=1 --with-fortran-bindings=0 --with-shared-libraries=1 --with-x=0 --with-zlib --download-openmpi=https://urldefense.us/v3/__https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.3rc1.tar.bz2__;!!G_uCfscf7eWS!aCLPUhLfLFG5UwNlUWgGGhXZlw905gJwDd >>AryIrltDIXJcdmOP6Is44FzBVrY5ndwzmIqMhI515mnNjTuHoR-tzq$ --download-pastix=https://urldefense.us/v3/__https://web.cels.anl.gov/projects/petsc/download/externalpackages/pastix_5.2.3-p1.tar.bz2__;!!G_uCfscf7eWS!aCLPUhLfLFG5UwNlUWgGGhXZlw905gJwDdAryIrltDIXJcdmOP6Is44FzBVrY5ndwzmIqMhI515mnNjTuA >>fJ49xl$ && make && make check >> >> There's an error encountered during configuration with the above options: >> ``` >> TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:676) >> * >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): >> - >> Fortran error! mpi_init() could not be located! >> * >> ``` >> Please refer to the attached file for further information. > > So I'm getting: > >>>>>>> > *** Fortran compiler > checking whether the compiler supports GNU Fortran... yes > checking whether gfortran accepts -g... yes > checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B > checking the name lister (/usr/bin/nm -B) interface... BSD nm > checking whether ln -s works... yes > checking if Fortran compiler works... yes > checking for extra arguments to build a shared library... impossible -- -static > checking for gfortran warnings flags... none > checking for Fortran flag to compile .f files... none > checking for Fortran flag to compile .f90 files... none > checking if Fortran compilers preprocess .F90 files without additional flag... yes > checking to see if Fortran compilers need additional linker flags... -Wl,-flat_namespace > checking external symbol convention... single underscore > checking if C and Fortran are link compatible... yes > checking to see if Fortran compiler likes the C++ exception flags... skipped (no C++ exceptions flags) > checking to see if mpifort compiler needs additional linker flags... none > <<<< > > However you are getting: > >>>>> > *** Fortran compiler > checking whether the compiler supports GNU Fortran... yes > checking whether gfortran accepts -g... yes > checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B > checking the name lister (/usr/bin/nm -B) interface... BSD nm > checking whether ln -s works... yes > checking if Fortran compiler works... yes > checking for extra arguments to build a shared library... impossible -- -static > checking for gfortran warnings flags... none > checking for Fortran flag to compile .f files... none > checking for Fortran flag to compile .f90 files... none > checking if Fortran compilers preprocess .F90 files without additional flag... yes > checking to see if Fortran compilers need additional linker flags... -Wl,-flat_namespace > checking external symbol convention... single underscore > checking if C and Fortran are link compatible... yes > checking to see if Fortran compiler likes the C++ exception flags... skipped (no C++ exceptions flags) > checking to see if mpifort compiler needs additional linker flags... -Wl,-commons,use_dylibs > <<<<
Re: [petsc-users] ex19: Segmentation Violation when run with MUMPS on MacOS (arm64)
> On 31 Mar 2024, at 02: 59, Satish Balay via petsc-users wrote: > > I'll just note - I can reproduce with: > > petsc@ npro petsc. x % ./configure --download-mpich --download-mumps --download-scalapack ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd > On 31 Mar 2024, at 02:59, Satish Balay via petsc-users wrote: > > I'll just note - I can reproduce with: > > petsc@npro petsc.x % ./configure --download-mpich --download-mumps --download-scalapack && make && make check > > And then - the following work fine for me: > > petsc@npro petsc.x % ./configure --download-mpich --download-mumps --download-scalapack COPTFLAGS=-O0 FOPTFLAGS=-O0 LDFLAGS=-Wl,-ld_classic && make && make check Hello, I've encountered the same issue. Even after building PETSc with the aforementioned configuration, the error persists. Apart from the segmentation fault, occasionally the following error arises: ``` > === > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 47531 RUNNING AT yzzs-mac.local > = EXIT CODE: 6 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > === > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Terminated: 15 (signal 15) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions /Users/zzyang/workspace/repos/petsc/src/snes/tutorials Possible problem with ex19 running with MUMPS, diffs above ``` Is there a possibility that this is related to the Fortran bindings of MPICH? Best wishes, Zongze > >CLINKER arch-darwin-c-debug/lib/libpetsc.3.021.0.dylib > DSYMUTIL arch-darwin-c-debug/lib/libpetsc.3.021.0.dylib > = > Now to check if the libraries are working do: > make PETSC_DIR=/Users/petsc/petsc.x PETSC_ARCH=arch-darwin-c-debug check > = > Running PETSc check examples to verify correct installation > Using PETSC_DIR=/Users/petsc/petsc.x and PETSC_ARCH=arch-darwin-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed PETSc check examples > petsc@npro petsc.x % > > petsc@npro petsc.z % ./configure --download-openmpi=https://urldefense.us/v3/__https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.3rc1.tar.bz2__;!!G_uCfscf7eWS!adsQx6TTtuIrMPz2ZorSv0iNPff5Fm5aFUi4i9n_E7v3GfAc8XCDtrlfVFyfPEgDFe4lKLL1XzkJ3c_97YoeukE$ --download-mumps --download-scalapack && make && make check > > DSYMUTIL arch-darwin-c-debug/lib/libpetsc.3.021.0.dylib > = > Now to check if the libraries are working do: > make PETSC_DIR=/Users/petsc/petsc.z PETSC_ARCH=arch-darwin-c-debug check > = > Running PETSc check examples to verify correct installation > Using PETSC_DIR=/Users/petsc/petsc.z and PETSC_ARCH=arch-darwin-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed PETSc check examples > petsc@npro petsc.z % > > [however parmmg and pastix are failing to build with openmpi] > > And I thought this worked for me yesterday - but I see failures now. > > ./configure --download-bison --download-chaco --download-ctetgen --download-eigen --download-fftw --download-hdf5 --download-hpddm --download-hwloc --download-hwloc-configure-arguments=--disable-opencl --download-hypre --download-libpng --download-metis --download-mmg --download-mpich --download-mpich-configure-arguments=--disable-opencl --download-mumps --download-netcdf --download-openblas --download-openblas-make-options="'USE_THREAD=0 USE_LOCKING=1 USE_OPENMP=0'" --download-parmmg --download-pastix --download-pnetcdf --download-pragmatic --download-ptscotch --download-scalapack --download-slepc --download-suitesparse --download-superlu_dist --download-tetgen --download-triangle --with-c2html=0 --with-debugging=1 --with-fortran-bindings=0 --with-shared-libraries=1 --with-x=0 --with-zlib --COPTFLAGS=-O0 --FOPTFLAGS=-O0 --LDFLAGS=-Wl,-ld_classic --with-clean > > Satish > > On Sat, 30 Mar 2024, Barry Smith wrote: > >> >> Can you check the value of IRHSCOMP in the debugger? Using gdb as the debugger may work better for this. >> >> Barry >> >> >>> On Mar
Re: [petsc-users] Install PETSc with option `--with-shared-libraries=1` failed on MacOS
The issue of openblas was resolved by this pr https://urldefense.us/v3/__https://github.com/OpenMathLib/OpenBLAS/pull/4565__;!!G_uCfscf7eWS!b09n5clcTFuLceLY_9KfqtSsgmmCIBLFbqciRVCKvnvFw9zTaNF8ssK0MiQlBOXUJe7H88nl-7ExdfhB-cMXLQ2d$ Best wishes, Zongze > On 18 Mar 2024, at 00:50, Zongze Yang wrote: > > It can be resolved by adding CFLAGS=-Wno-int-conversion. Perhaps the default > behaviour of the new version compiler has been changed? > > Best wishes, > Zongze >> On 18 Mar 2024, at 00:23, Satish Balay wrote: >> >> Hm - I just tried a build with balay/xcode15-mpich - and that goes through >> fine for me. So don't know what the difference here is. >> >> One difference is - I have a slightly older xcode. However your compiler >> appears to behave as using -Werror. Perhaps CFLAGS=-Wno-int-conversion will >> help here? >> >> Satish >> >> >> Executing: gcc --version >> stdout: >> Apple clang version 15.0.0 (clang-1500.3.9.4) >> >> Executing: /Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/bin/mpicc >> -show >> stdout: gcc -fPIC -fno-stack-check -Qunused-arguments -g -O0 >> -Wno-implicit-function-declaration -fno-common >> -I/Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/include >> -L/Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/lib -lmpi -lpmpi >> >> /Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/bin/mpicc -O2 >> -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_WARMUP >> -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 >> -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.21\" >> -march=armv8-a -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME >> -DASMNAME=_lapack_wrappers -DASMFNAME=_lapack_wrappers_ >> -DNAME=lapack_wrappers_ -DCNAME=lapack_wrappers >> -DCHAR_NAME=\"lapack_wrappers_\" -DCHAR_CNAME=\"lapack_wrappers\" >> -DNO_AFFINITY -I.. -c src/lapack_wrappers.c -o src/lapack_wrappers.o >> src/lapack_wrappers.c:570:81: error: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >>RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, beta, >> C, info); >> >> ^~~~ >> >> & >> >> vs: >> Executing: gcc --version >> stdout: >> Apple clang version 15.0.0 (clang-1500.1.0.2.5) >> >> Executing: /Users/balay/petsc/arch-darwin-c-debug/bin/mpicc -show >> stdout: gcc -fPIC -fno-stack-check -Qunused-arguments -g -O0 >> -Wno-implicit-function-declaration -fno-common >> -I/Users/balay/petsc/arch-darwin-c-debug/include >> -L/Users/balay/petsc/arch-darwin-c-debug/lib -lmpi -lpmpi >> >> >> /Users/balay/petsc/arch-darwin-c-debug/bin/mpicc -O2 -DMAX_STACK_ALLOC=2048 >> -Wall -DF_INTERFACE_GFORT -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=24 >> -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 >> -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.21\" -march=armv8-a -UASMNAME -UASMFNAME >> -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_lapack_wrappers >> -DASMFNAME=_lapack_wrappers_ -DNAME=lapack_wrappers_ -DCNAME=lapack_wrappers >> -DCHAR_NAME=\"lapack_wrappers_\" -DCHAR_CNAME=\"lapack_wrappers\" >> -DNO_AFFINITY -I.. -c src/lapack_wrappers.c -o src/lapack_wrappers.o >> src/lapack_wrappers.c:570:81: warning: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >>RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, beta, >> C, info); >> >> ^~~~ >> >> & >> >> >> >> >> On Sun, 17 Mar 2024, Pierre Jolivet wrote: >> >>> Ah, my bad, I misread linux-opt-arm as a macOS runner, no wonder the option >>> is not helping… >>> Take Barry’s advice. >>> Furthermore, it looks like OpenBLAS people are steering in the opposite >>> direction as us, by forcing the use of ld-classic >>> https://urldefense.us/v3/__https://github.com/OpenMathLib/OpenBLAS/commit/103d6f4e42fbe532ae4ea48e8d90d7d792
Re: [petsc-users] Install PETSc with option `--with-shared-libraries=1` failed on MacOS
It can be resolved by adding CFLAGS=-Wno-int-conversion. Perhaps the default behaviour of the new version compiler has been changed? Best wishes, Zongze > On 18 Mar 2024, at 00:23, Satish Balay wrote: > > Hm - I just tried a build with balay/xcode15-mpich - and that goes through > fine for me. So don't know what the difference here is. > > One difference is - I have a slightly older xcode. However your compiler > appears to behave as using -Werror. Perhaps CFLAGS=-Wno-int-conversion will > help here? > > Satish > > > Executing: gcc --version > stdout: > Apple clang version 15.0.0 (clang-1500.3.9.4) > > Executing: /Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/bin/mpicc > -show > stdout: gcc -fPIC -fno-stack-check -Qunused-arguments -g -O0 > -Wno-implicit-function-declaration -fno-common > -I/Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/include > -L/Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/lib -lmpi -lpmpi > > /Users/zzyang/workspace/repos/petsc/arch-darwin-c-debug/bin/mpicc -O2 > -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_WARMUP > -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 > -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.21\" -march=armv8-a > -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME > -DASMNAME=_lapack_wrappers -DASMFNAME=_lapack_wrappers_ > -DNAME=lapack_wrappers_ -DCNAME=lapack_wrappers > -DCHAR_NAME=\"lapack_wrappers_\" -DCHAR_CNAME=\"lapack_wrappers\" > -DNO_AFFINITY -I.. -c src/lapack_wrappers.c -o src/lapack_wrappers.o > src/lapack_wrappers.c:570:81: error: incompatible integer to pointer > conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint > *' (aka 'const int *'); take the address with & [-Wint-conversion] >RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, beta, > C, info); > > ^~~~ > > & > > vs: > Executing: gcc --version > stdout: > Apple clang version 15.0.0 (clang-1500.1.0.2.5) > > Executing: /Users/balay/petsc/arch-darwin-c-debug/bin/mpicc -show > stdout: gcc -fPIC -fno-stack-check -Qunused-arguments -g -O0 > -Wno-implicit-function-declaration -fno-common > -I/Users/balay/petsc/arch-darwin-c-debug/include > -L/Users/balay/petsc/arch-darwin-c-debug/lib -lmpi -lpmpi > > > /Users/balay/petsc/arch-darwin-c-debug/bin/mpicc -O2 -DMAX_STACK_ALLOC=2048 > -Wall -DF_INTERFACE_GFORT -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=24 > -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 > -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.21\" -march=armv8-a -UASMNAME -UASMFNAME > -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=_lapack_wrappers > -DASMFNAME=_lapack_wrappers_ -DNAME=lapack_wrappers_ -DCNAME=lapack_wrappers > -DCHAR_NAME=\"lapack_wrappers_\" -DCHAR_CNAME=\"lapack_wrappers\" > -DNO_AFFINITY -I.. -c src/lapack_wrappers.c -o src/lapack_wrappers.o > src/lapack_wrappers.c:570:81: warning: incompatible integer to pointer > conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint > *' (aka 'const int *'); take the address with & [-Wint-conversion] >RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, beta, > C, info); > > ^~~~ > > & > > > > > On Sun, 17 Mar 2024, Pierre Jolivet wrote: > >> Ah, my bad, I misread linux-opt-arm as a macOS runner, no wonder the option >> is not helping… >> Take Barry’s advice. >> Furthermore, it looks like OpenBLAS people are steering in the opposite >> direction as us, by forcing the use of ld-classic >> https://urldefense.us/v3/__https://github.com/OpenMathLib/OpenBLAS/commit/103d6f4e42fbe532ae4ea48e8d90d7d792bc93d2__;!!G_uCfscf7eWS!bY2l3X9Eb5PRzNQYrfPFXhgcUodHCiDinhQYga0PeQn1IQzJYD376fk-pZfktGAkpTvBmzy7BFDc9SrazFoooQ$ >> , so that’s another good argument in favor of -framework Accelerate. >> >> Thanks, >> Pierre >> >> PS: anyone benchmarked those >> https://urldefense.us/v3/__https://developer.apple.com/documentation/accelerate/sparse_solvers__;!!G_uCfscf7eWS!bY2l3X9Eb5PRzNQYrfPFXhgcUodHCiDinhQYga0PeQn1IQzJYD376fk-pZfktGAkpTvBmzy7BFDc9SrpnDvT5g$ >> ? I didn’t even know they existed. >> >>> On 17 Mar 2024, at 3:06 PM, Zongze Yang >> <mailto:yangzon...@gmail.com>> wrote: >>>
Re: [petsc-users] Install PETSc with option `--with-shared-libraries=1` failed on MacOS
Understood. Thank you for your advice. Best wishes, Zongze > On 17 Mar 2024, at 22:04, Barry Smith wrote: > > > I would just avoid the --download-openblas option. The BLAS/LAPACK > provided by Apple should perform fine, perhaps even better than OpenBLAS on > your system. > > >> On Mar 17, 2024, at 9:58 AM, Zongze Yang wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Adding the flag `--download-openblas-make-options=TARGET=GENERIC` did not >> resolve the issue. The same error persisted. >> >> Best wishes, >> Zongze >> >>> On 17 Mar 2024, at 20:58, Pierre Jolivet >> <mailto:pie...@joliv.et>> wrote: >>> >>> >>> >>>> On 17 Mar 2024, at 1:04 PM, Zongze Yang >>> <mailto:yangzon...@gmail.com>> wrote: >>>> >>>> Thank you for providing the instructions. I try the first option. >>>> >>>> Now, the error of the configuration is related to OpenBLAS. >>>> Add `--CFLAGS=-Wno-int-conversion` to configure command resolve this. >>>> Should this be reported to OpenBLAS? Or need to fix the configure in petsc? >>> >>> I see our linux-opt-arm runner is using the additional flag >>> '--download-openblas-make-options=TARGET=GENERIC', could you maybe try to >>> add that as well? >>> I don’t think there is much to fix on our end, OpenBLAS has been very >>> broken lately on arm (current version is 0.3.26 but we can’t update because >>> there is a huge performance regression which makes the pipeline timeout). >>> >>> Thanks, >>> Pierre >>> >>>> >>>> The configure.log is attached. The errors are show below: >>>> ``` >>>> src/lapack_wrappers.c:570:81: error: incompatible integer to pointer >>>> conversion passing 'blasint' (aka 'int') to parameter of type 'const >>>> blasint *' (aka 'const int *'); take the address with & [-Wint-conversion] >>>> RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >>>> beta, C, info); >>>> >>>> ^~~~ >>>> >>>> & >>>> src/../inc/relapack.h:74:216: note: passing argument to parameter here >>>> void RELAPACK_sgemmt(const char *, const char *, const char *, const >>>> blasint *, const blasint *, const float *, const float *, const blasint *, >>>> const float *, const blasint *, const float *, float *, const blasint *); >>>> >>>> >>>> ^ >>>> src/lapack_wrappers.c:583:81: error: incompatible integer to pointer >>>> conversion passing 'blasint' (aka 'int') to parameter of type 'const >>>> blasint *' (aka 'const int *'); take the address with & [-Wint-conversion] >>>> RELAPACK_dgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >>>> beta, C, info); >>>> >>>> ^~~~ >>>> >>>> & >>>> src/../inc/relapack.h:75:221: note: passing argument to parameter here >>>> void RELAPACK_dgemmt(const char *, const char *, const char *, const >>>> blasint *, const blasint *, const double *, const double *, const blasint >>>> *, const double *, const blasint *, const double *, double *, const >>>> blasint *); >>>> >>>> >>>> ^ >>>> src/lapack_wrappers.c:596:81: error: incompatible integer to pointer >>>> conversion passing 'blasint' (aka 'int') to parameter of type 'const >>>> blasint *' (aka 'const int *'); take the address with & [-Wint-conversion] >>>> RELAPACK_cgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >>>> beta, C, info); >>
Re: [petsc-users] Install PETSc with option `--with-shared-libraries=1` failed on MacOS
Adding the flag `--download-openblas-make-options=TARGET=GENERIC` did not resolve the issue. The same error persisted. Best wishes, Zongze > On 17 Mar 2024, at 20:58, Pierre Jolivet wrote: > > > >> On 17 Mar 2024, at 1:04 PM, Zongze Yang wrote: >> >> Thank you for providing the instructions. I try the first option. >> >> Now, the error of the configuration is related to OpenBLAS. >> Add `--CFLAGS=-Wno-int-conversion` to configure command resolve this. Should >> this be reported to OpenBLAS? Or need to fix the configure in petsc? > > I see our linux-opt-arm runner is using the additional flag > '--download-openblas-make-options=TARGET=GENERIC', could you maybe try to add > that as well? > I don’t think there is much to fix on our end, OpenBLAS has been very broken > lately on arm (current version is 0.3.26 but we can’t update because there is > a huge performance regression which makes the pipeline timeout). > > Thanks, > Pierre > >> >> The configure.log is attached. The errors are show below: >> ``` >> src/lapack_wrappers.c:570:81: error: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >> RELAPACK_sgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >> beta, C, info); >> >>^~~~ >> >>& >> src/../inc/relapack.h:74:216: note: passing argument to parameter here >> void RELAPACK_sgemmt(const char *, const char *, const char *, const >> blasint *, const blasint *, const float *, const float *, const blasint *, >> const float *, const blasint *, const float *, float *, const blasint *); >> >> >> ^ >> src/lapack_wrappers.c:583:81: error: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >> RELAPACK_dgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >> beta, C, info); >> >>^~~~ >> >>& >> src/../inc/relapack.h:75:221: note: passing argument to parameter here >> void RELAPACK_dgemmt(const char *, const char *, const char *, const >> blasint *, const blasint *, const double *, const double *, const blasint *, >> const double *, const blasint *, const double *, double *, const blasint *); >> >> >> ^ >> src/lapack_wrappers.c:596:81: error: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >> RELAPACK_cgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >> beta, C, info); >> >>^~~~ >> >>& >> src/../inc/relapack.h:76:216: note: passing argument to parameter here >> void RELAPACK_cgemmt(const char *, const char *, const char *, const >> blasint *, const blasint *, const float *, const float *, const blasint *, >> const float *, const blasint *, const float *, float *, const blasint *); >> >> >> ^ >> src/lapack_wrappers.c:609:81: error: incompatible integer to pointer >> conversion passing 'blasint' (aka 'int') to parameter of type 'const blasint >> *' (aka 'const int *'); take the address with & [-Wint-conversion] >> RELAPACK_zgemmt(uplo, transA, transB, n, k, alpha, A, ldA, B, ldB, >> beta, C, info)
Re: [petsc-users] Help Needed Debugging Installation Issue for PETSc with SLEPc
Thanks very much! Best wishes, Zongze > On 15 Mar 2024, at 22:50, Pierre Jolivet wrote: > > This was fixed 5 days ago in > https://urldefense.us/v3/__https://gitlab.com/slepc/slepc/-/merge_requests/638__;!!G_uCfscf7eWS!fPbzaiH98Qdihq74E8kcX8JQE_EGk_PhMqYnheGPjwJD2l8AhgXg3ZrEkFZmV0lfG8vVlKawhAaQq4eSMRZVffeF$ > , so you need to use an up-to-date release branch of SLEPc. > > Thanks, > Pierre > >> On 15 Mar 2024, at 3:44 PM, Zongze Yang wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hi, >> >> I am currently facing an issue while attempting to install PETSc with SLEPc. >> Despite not encountering any errors in the log generated by the 'make' >> command, I am receiving an error message stating "Error during compile". >> >> I would greatly appreciate it if someone could provide me with some guidance >> on debugging this issue. >> >> I have attached the configure logs and make logs for both PETSc and SLEPc >> for your reference. >> >> Below is an excerpt from the make.log file of SLEPc: >> ``` >> CLINKER default/lib/libslepc.3.020.1.dylib >> ld: warning: -commons use_dylibs is no longer supported, using error >> treatment instead >> ld: warning: -commons use_dylibs is no longer supported, using error >> treatment instead >> ld: warning: -commons use_dylibs is no longer supported, using error >> treatment instead >> ld: warning: duplicate -rpath >> '/Users/zzyang/opt/firedrake/firedrake-real-int32-debug/src/petsc/default/lib' >> ignored >> ld: warning: dylib >> (/opt/homebrew/Cellar/gcc/13.2.0/lib/gcc/current/libgfortran.dylib) was >> built for newer macOS version (14.0) than being linked (12.0) >> ld: warning: dylib >> (/opt/homebrew/Cellar/gcc/13.2.0/lib/gcc/current/libquadmath.dylib) was >> built for newer macOS version (14.0) than being linked (12.0) >>DSYMUTIL default/lib/libslepc.3.020.1.dylib >> gmake[6]: Leaving directory >> '/Users/zzyang/opt/firedrake/firedrake-real-int32-debug/src/petsc/default/externalpackages/git.slepc' >> gmake[5]: Leaving directory >> '/Users/zzyang/opt/firedrake/firedrake-real-int32-debug/src/petsc/default/externalpackages/git.slepc' >> ***ERROR >> Error during compile, check default/lib/slepc/conf/make.log >> Send all contents of ./default/lib/slepc/conf to slepc-ma...@upv.es >> <mailto:slepc-ma...@upv.es> >> >> Finishing make run at 五, 15 3 2024 21:04:17 +0800 >> ``` >> >> Thank you very much for your attention and support. >> >> Best wishes, >> Zongze >> >> >
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
On Tue, 23 May 2023 at 20:09, Yann Jobic wrote: > If i may, you can use the command line option "-mat_mumps_icntl_4 2" > MUMPS then gives infomations about the factorization step, such as the > estimated needed memory. > > Thank you for your suggestion! Best wishes, Zongze Best regards, > > Yann > > Le 5/23/2023 à 11:59 AM, Matthew Knepley a écrit : > > On Mon, May 22, 2023 at 10:42 PM Zongze Yang > <mailto:yangzon...@gmail.com>> wrote: > > > > On Tue, 23 May 2023 at 05:31, Stefano Zampini > > mailto:stefano.zamp...@gmail.com>> > wrote: > > > > If I may add to the discussion, it may be that you are going OOM > > since you are trying to factorize a 3 million dofs problem, this > > problem goes undetected and then fails at a later stage > > > > Thank you for your comment. I ran the problem with 90 processes > > distributed across three nodes, each equipped with 500G of memory. > > If this amount of memory is sufficient for solving the matrix with > > approximately 3 million degrees of freedom? > > > > > > It really depends on the fill. Suppose that you get 1% fill, then > > > >(3e6)^2 * 0.01 * 8 = 1e12 B > > > > and you have 1.5e12 B, so I could easily see running out of memory. > > > >Thanks, > > > > Matt > > > > Thanks! > > Zongze > > > > Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang > > mailto:yangzon...@gmail.com>> ha scritto: > > > > Thanks! > > > > Zongze > > > > Matthew Knepley > <mailto:knep...@gmail.com>>于2023年5月23日 周二00:09写道： > > > > On Mon, May 22, 2023 at 11:07 AM Zongze Yang > > mailto:yangzon...@gmail.com>> > wrote: > > > > Hi, > > > > I hope this letter finds you well. I am writing to > > seek guidance regarding an error I encountered while > > solving a matrix using MUMPS on multiple nodes: > > > > > > Iprobe is buggy on several MPI implementations. PETSc > > has an option for shutting it off for this reason. > > I do not know how to shut it off inside MUMPS however. I > > would mail their mailing list to see. > > > >Thanks, > > > > Matt > > > > ```bash > > Abort(1681039) on node 60 (rank 60 in comm 240): > > Fatal error in PMPI_Iprobe: Other MPI error, error > > stack: > > PMPI_Iprobe(124)..: > > MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, > > comm=0xc426, flag=0x7ffc130f9c4c, > > status=0x7ffc130f9e80) failed > > MPID_Iprobe(240)..: > > MPIDI_iprobe_safe(108): > > MPIDI_iprobe_unsafe(35)...: > > MPIDI_OFI_do_iprobe(69)...: > > MPIDI_OFI_handle_cq_error(949): OFI poll failed > > > (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) > > Assertion failed in file > > src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 > > ``` > > > > The matrix in question has a degree of freedom (dof) > > of 3.86e+06. Interestingly, when solving > > smaller-scale problems, everything functions > > perfectly without any issues. However, when > > attempting to solve the larger matrix on multiple > > nodes, I encounter the aforementioned error. > > > > The complete error message I received is as follows: > > ```bash > > Abort(1681039) on node 60 (rank 60 in comm 240): > > Fatal error in PMPI_Iprobe: Other MPI error, error > > stack: > > PMPI_Iprobe(124)..: > > MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, > > comm=0xc426, flag=0x7ffc130f9c4c, > > status=0x7ffc130f9e80) failed > > MPID_Iprobe(240)..: > >
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
On Tue, 23 May 2023 at 19:51, Zongze Yang wrote: > Thank you for your suggestion. I solved the problem with SuperLU_DIST, and > it works well. > This is solved with four nodes, each equipped with 500G of memory. Best wishes, Zongze Best wishes, > Zongze > > > On Tue, 23 May 2023 at 18:00, Matthew Knepley wrote: > >> On Mon, May 22, 2023 at 10:46 PM Zongze Yang >> wrote: >> >>> I have an additional question to ask: Is it possible for the >>> SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) >>> as MUMPS? >>> >> >> I do not know if they use that function. But it is easy to try it out, so >> I would. >> >> Thanks, >> >> Matt >> >> >>> Best wishes, >>> Zongze >>> >>> >>> On Tue, 23 May 2023 at 10:41, Zongze Yang wrote: >>> >>>> On Tue, 23 May 2023 at 05:31, Stefano Zampini < >>>> stefano.zamp...@gmail.com> wrote: >>>> >>>>> If I may add to the discussion, it may be that you are going OOM since >>>>> you are trying to factorize a 3 million dofs problem, this problem goes >>>>> undetected and then fails at a later stage >>>>> >>>> >>>> Thank you for your comment. I ran the problem with 90 processes >>>> distributed across three nodes, each equipped with 500G of memory. If this >>>> amount of memory is sufficient for solving the matrix with approximately 3 >>>> million degrees of freedom? >>>> >>>> Thanks! >>>> Zongze >>>> >>>> Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang < >>>>> yangzon...@gmail.com> ha scritto: >>>>> >>>>>> Thanks! >>>>>> >>>>>> Zongze >>>>>> >>>>>> Matthew Knepley 于2023年5月23日 周二00:09写道： >>>>>> >>>>>>> On Mon, May 22, 2023 at 11:07 AM Zongze Yang >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I hope this letter finds you well. I am writing to seek guidance >>>>>>>> regarding an error I encountered while solving a matrix using MUMPS on >>>>>>>> multiple nodes: >>>>>>>> >>>>>>> >>>>>>> Iprobe is buggy on several MPI implementations. PETSc has an option >>>>>>> for shutting it off for this reason. >>>>>>> I do not know how to shut it off inside MUMPS however. I would mail >>>>>>> their mailing list to see. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> ```bash >>>>>>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>>>>>> PMPI_Iprobe: Other MPI error, error stack: >>>>>>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>>>>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>>>>>> status=0x7ffc130f9e80) failed >>>>>>>> MPID_Iprobe(240)..: >>>>>>>> MPIDI_iprobe_safe(108): >>>>>>>> MPIDI_iprobe_unsafe(35)...: >>>>>>>> MPIDI_OFI_do_iprobe(69)...: >>>>>>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>>>>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>>>>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at >>>>>>>> line 125: 0 >>>>>>>> ``` >>>>>>>> >>>>>>>> The matrix in question has a degree of freedom (dof) of 3.86e+06. >>>>>>>> Interestingly, when solving smaller-scale problems, everything >>>>>>>> functions >>>>>>>> perfectly without any issues. However, when attempting to solve the >>>>>>>> larger >>>>>>>> matrix on multiple nodes, I encounter the aforementioned error. >>>>>>>> >>>>>>>> The complete error message I received is as follows: >>>>>>>> ```bash >>>>>>>> Abort(1681039) on node 60 (rank 60 in comm
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
Thank you for your suggestion. I solved the problem with SuperLU_DIST, and it works well. Best wishes, Zongze On Tue, 23 May 2023 at 18:00, Matthew Knepley wrote: > On Mon, May 22, 2023 at 10:46 PM Zongze Yang wrote: > >> I have an additional question to ask: Is it possible for the SuperLU_DIST >> library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS? >> > > I do not know if they use that function. But it is easy to try it out, so > I would. > > Thanks, > > Matt > > >> Best wishes, >> Zongze >> >> >> On Tue, 23 May 2023 at 10:41, Zongze Yang wrote: >> >>> On Tue, 23 May 2023 at 05:31, Stefano Zampini >>> wrote: >>> >>>> If I may add to the discussion, it may be that you are going OOM since >>>> you are trying to factorize a 3 million dofs problem, this problem goes >>>> undetected and then fails at a later stage >>>> >>> >>> Thank you for your comment. I ran the problem with 90 processes >>> distributed across three nodes, each equipped with 500G of memory. If this >>> amount of memory is sufficient for solving the matrix with approximately 3 >>> million degrees of freedom? >>> >>> Thanks! >>> Zongze >>> >>> Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang < >>>> yangzon...@gmail.com> ha scritto: >>>> >>>>> Thanks! >>>>> >>>>> Zongze >>>>> >>>>> Matthew Knepley 于2023年5月23日 周二00:09写道： >>>>> >>>>>> On Mon, May 22, 2023 at 11:07 AM Zongze Yang >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I hope this letter finds you well. I am writing to seek guidance >>>>>>> regarding an error I encountered while solving a matrix using MUMPS on >>>>>>> multiple nodes: >>>>>>> >>>>>> >>>>>> Iprobe is buggy on several MPI implementations. PETSc has an option >>>>>> for shutting it off for this reason. >>>>>> I do not know how to shut it off inside MUMPS however. I would mail >>>>>> their mailing list to see. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> ```bash >>>>>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>>>>> PMPI_Iprobe: Other MPI error, error stack: >>>>>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>>>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>>>>> status=0x7ffc130f9e80) failed >>>>>>> MPID_Iprobe(240)..: >>>>>>> MPIDI_iprobe_safe(108): >>>>>>> MPIDI_iprobe_unsafe(35)...: >>>>>>> MPIDI_OFI_do_iprobe(69)...: >>>>>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>>>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>>>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at >>>>>>> line 125: 0 >>>>>>> ``` >>>>>>> >>>>>>> The matrix in question has a degree of freedom (dof) of 3.86e+06. >>>>>>> Interestingly, when solving smaller-scale problems, everything functions >>>>>>> perfectly without any issues. However, when attempting to solve the >>>>>>> larger >>>>>>> matrix on multiple nodes, I encounter the aforementioned error. >>>>>>> >>>>>>> The complete error message I received is as follows: >>>>>>> ```bash >>>>>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>>>>> PMPI_Iprobe: Other MPI error, error stack: >>>>>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>>>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>>>>> status=0x7ffc130f9e80) failed >>>>>>> MPID_Iprobe(240)..: >>>>>>> MPIDI_iprobe_safe(108): >>>>>>> MPIDI_iprobe_unsafe(35)...: >>>>>>> MPIDI_OFI_do_iprobe(69)...: >>>>>>> MPIDI_OFI_handle_cq_error(949)
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
I have an additional question to ask: Is it possible for the SuperLU_DIST library to encounter the same MPI problem (PMPI_Iprobe failed) as MUMPS? Best wishes, Zongze On Tue, 23 May 2023 at 10:41, Zongze Yang wrote: > On Tue, 23 May 2023 at 05:31, Stefano Zampini > wrote: > >> If I may add to the discussion, it may be that you are going OOM since >> you are trying to factorize a 3 million dofs problem, this problem goes >> undetected and then fails at a later stage >> > > Thank you for your comment. I ran the problem with 90 processes > distributed across three nodes, each equipped with 500G of memory. If this > amount of memory is sufficient for solving the matrix with approximately 3 > million degrees of freedom? > > Thanks! > Zongze > > Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang >> ha scritto: >> >>> Thanks! >>> >>> Zongze >>> >>> Matthew Knepley 于2023年5月23日 周二00:09写道： >>> >>>> On Mon, May 22, 2023 at 11:07 AM Zongze Yang >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I hope this letter finds you well. I am writing to seek guidance >>>>> regarding an error I encountered while solving a matrix using MUMPS on >>>>> multiple nodes: >>>>> >>>> >>>> Iprobe is buggy on several MPI implementations. PETSc has an option for >>>> shutting it off for this reason. >>>> I do not know how to shut it off inside MUMPS however. I would mail >>>> their mailing list to see. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> ```bash >>>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>>> PMPI_Iprobe: Other MPI error, error stack: >>>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>>> status=0x7ffc130f9e80) failed >>>>> MPID_Iprobe(240)..: >>>>> MPIDI_iprobe_safe(108): >>>>> MPIDI_iprobe_unsafe(35)...: >>>>> MPIDI_OFI_do_iprobe(69)...: >>>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >>>>> 125: 0 >>>>> ``` >>>>> >>>>> The matrix in question has a degree of freedom (dof) of 3.86e+06. >>>>> Interestingly, when solving smaller-scale problems, everything functions >>>>> perfectly without any issues. However, when attempting to solve the larger >>>>> matrix on multiple nodes, I encounter the aforementioned error. >>>>> >>>>> The complete error message I received is as follows: >>>>> ```bash >>>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>>> PMPI_Iprobe: Other MPI error, error stack: >>>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>>> status=0x7ffc130f9e80) failed >>>>> MPID_Iprobe(240)..: >>>>> MPIDI_iprobe_safe(108): >>>>> MPIDI_iprobe_unsafe(35)...: >>>>> MPIDI_OFI_do_iprobe(69)...: >>>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >>>>> 125: 0 >>>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) >>>>> [0x7f6076063f2c] >>>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) >>>>> [0x7f6075fc5c24] >>>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) >>>>> [0x7f6076044c51] >>>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) >>>>> [0x7f6076047799] >>>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
On Tue, 23 May 2023 at 05:31, Stefano Zampini wrote: > If I may add to the discussion, it may be that you are going OOM since you > are trying to factorize a 3 million dofs problem, this problem goes > undetected and then fails at a later stage > Thank you for your comment. I ran the problem with 90 processes distributed across three nodes, each equipped with 500G of memory. If this amount of memory is sufficient for solving the matrix with approximately 3 million degrees of freedom? Thanks! Zongze Il giorno lun 22 mag 2023 alle ore 20:03 Zongze Yang > ha scritto: > >> Thanks! >> >> Zongze >> >> Matthew Knepley 于2023年5月23日 周二00:09写道： >> >>> On Mon, May 22, 2023 at 11:07 AM Zongze Yang >>> wrote: >>> >>>> Hi, >>>> >>>> I hope this letter finds you well. I am writing to seek guidance >>>> regarding an error I encountered while solving a matrix using MUMPS on >>>> multiple nodes: >>>> >>> >>> Iprobe is buggy on several MPI implementations. PETSc has an option for >>> shutting it off for this reason. >>> I do not know how to shut it off inside MUMPS however. I would mail >>> their mailing list to see. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> ```bash >>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>> PMPI_Iprobe: Other MPI error, error stack: >>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>> status=0x7ffc130f9e80) failed >>>> MPID_Iprobe(240)..: >>>> MPIDI_iprobe_safe(108): >>>> MPIDI_iprobe_unsafe(35)...: >>>> MPIDI_OFI_do_iprobe(69)...: >>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >>>> 125: 0 >>>> ``` >>>> >>>> The matrix in question has a degree of freedom (dof) of 3.86e+06. >>>> Interestingly, when solving smaller-scale problems, everything functions >>>> perfectly without any issues. However, when attempting to solve the larger >>>> matrix on multiple nodes, I encounter the aforementioned error. >>>> >>>> The complete error message I received is as follows: >>>> ```bash >>>> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >>>> PMPI_Iprobe: Other MPI error, error stack: >>>> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >>>> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >>>> status=0x7ffc130f9e80) failed >>>> MPID_Iprobe(240)..: >>>> MPIDI_iprobe_safe(108): >>>> MPIDI_iprobe_unsafe(35)...: >>>> MPIDI_OFI_do_iprobe(69)...: >>>> MPIDI_OFI_handle_cq_error(949): OFI poll failed >>>> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >>>> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >>>> 125: 0 >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) >>>> [0x7f6076063f2c] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) >>>> [0x7f6075fc5c24] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) >>>> [0x7f6076044c51] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) >>>> [0x7f6076047799] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) >>>> [0x7f6075ff9e18] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) >>>> [0x7f6075ffa272] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) >>>> [0x7f6075e76836] >>>> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) >>>> [0x7f6075e7690d] >&g
Re: [petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
Thanks! Zongze Matthew Knepley 于2023年5月23日 周二00:09写道： > On Mon, May 22, 2023 at 11:07 AM Zongze Yang wrote: > >> Hi, >> >> I hope this letter finds you well. I am writing to seek guidance >> regarding an error I encountered while solving a matrix using MUMPS on >> multiple nodes: >> > > Iprobe is buggy on several MPI implementations. PETSc has an option for > shutting it off for this reason. > I do not know how to shut it off inside MUMPS however. I would mail their > mailing list to see. > > Thanks, > > Matt > > >> ```bash >> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >> PMPI_Iprobe: Other MPI error, error stack: >> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >> status=0x7ffc130f9e80) failed >> MPID_Iprobe(240)..: >> MPIDI_iprobe_safe(108): >> MPIDI_iprobe_unsafe(35)...: >> MPIDI_OFI_do_iprobe(69)...: >> MPIDI_OFI_handle_cq_error(949): OFI poll failed >> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >> 125: 0 >> ``` >> >> The matrix in question has a degree of freedom (dof) of 3.86e+06. >> Interestingly, when solving smaller-scale problems, everything functions >> perfectly without any issues. However, when attempting to solve the larger >> matrix on multiple nodes, I encounter the aforementioned error. >> >> The complete error message I received is as follows: >> ```bash >> Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in >> PMPI_Iprobe: Other MPI error, error stack: >> PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, >> tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, >> status=0x7ffc130f9e80) failed >> MPID_Iprobe(240)..: >> MPIDI_iprobe_safe(108): >> MPIDI_iprobe_unsafe(35)...: >> MPIDI_OFI_do_iprobe(69)...: >> MPIDI_OFI_handle_cq_error(949): OFI poll failed >> (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) >> Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line >> 125: 0 >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) >> [0x7f6076063f2c] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) >> [0x7f6075fc5c24] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) >> [0x7f6076044c51] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) >> [0x7f6076047799] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) >> [0x7f6075ff9e18] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) >> [0x7f6075ffa272] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) >> [0x7f6075e76836] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) >> [0x7f6075e7690d] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) >> [0x7f607602937b] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) >> [0x7f6075ff5471] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) >> [0x7f6075fafacd] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) >> [0x7f6075fafbea] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) >> [0x7f6075ddd542] >> /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) >> [0x7f606e08f19f] >> /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) >> [
[petsc-users] MPI_Iprobe Error with MUMPS Solver on Multi-Nodes
Hi, I hope this letter finds you well. I am writing to seek guidance regarding an error I encountered while solving a matrix using MUMPS on multiple nodes: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..: MPIDI_iprobe_safe(108): MPIDI_iprobe_unsafe(35)...: MPIDI_OFI_do_iprobe(69)...: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 ``` The matrix in question has a degree of freedom (dof) of 3.86e+06. Interestingly, when solving smaller-scale problems, everything functions perfectly without any issues. However, when attempting to solve the larger matrix on multiple nodes, I encounter the aforementioned error. The complete error message I received is as follows: ```bash Abort(1681039) on node 60 (rank 60 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc426, flag=0x7ffc130f9c4c, status=0x7ffc130f9e80) failed MPID_Iprobe(240)..: MPIDI_iprobe_safe(108): MPIDI_iprobe_unsafe(35)...: MPIDI_OFI_do_iprobe(69)...: MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error) Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_events.c at line 125: 0 /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPL_backtrace_show+0x26) [0x7f6076063f2c] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x41dc24) [0x7f6075fc5c24] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49cc51) [0x7f6076044c51] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x49f799) [0x7f6076047799] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x451e18) [0x7f6075ff9e18] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x452272) [0x7f6075ffa272] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce836) [0x7f6075e76836] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x2ce90d) [0x7f6075e7690d] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x48137b) [0x7f607602937b] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x44d471) [0x7f6075ff5471] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(+0x407acd) [0x7f6075fafacd] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPIR_Err_return_comm+0x10a) [0x7f6075fafbea] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpi.so.12(MPI_Iprobe+0x312) [0x7f6075ddd542] /nfs/opt/cascadelake/linux-centos7-cascadelake/gcc-9.4.0/mpich-3.4.2-qgtz76gekvjzuacy7wq5a26rqlewoxfc/lib/libmpifort.so.12(pmpi_iprobe+0x2f) [0x7f606e08f19f] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_load_MOD_zmumps_load_recv_msgs+0x142) [0x7f60737b194d] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_try_recvtreat_+0x34) [0x7f60738ab735] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(__zmumps_fac_par_m_MOD_zmumps_fac_par+0x991) [0x7f607378bcc8] /nfs/home/zzyang/opt/software/linux-centos7-cascadelake/gcc-9.4.0/mumps-5.5.1-gb7wlwxwbalf5rw5vkp6gtkhfkdqpntz/lib/libzmumps.so(zmumps_fac_par_i_+0x240) [0x7f6073881d36] Abort(805938831) on node 51 (rank 51 in comm 240): Fatal error in PMPI_Iprobe: Other MPI error, error stack: PMPI_Iprobe(124)..: MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, comm=0xc417, flag=0x7ffe20e1402c, status=0x7ffe20e14260) failed MPID_Iprobe(244)..: progress_test(100): MPIDI_OFI_handle_cq_error(949): OFI poll failed (ofi_events.c:951:MPIDI_OFI_handle_cq_error:Input/output error)
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Got it. Thank you for your explanation! Best wishes, Zongze On Mon, 15 May 2023 at 23:28, Matthew Knepley wrote: > On Mon, May 15, 2023 at 9:55 AM Zongze Yang wrote: > >> On Mon, 15 May 2023 at 17:24, Matthew Knepley wrote: >> >>> On Sun, May 14, 2023 at 7:23 PM Zongze Yang >>> wrote: >>> >>>> Could you try to project the coordinates into the continuity space by >>>> enabling the option >>>> `-dm_plex_gmsh_project_petscdualspace_lagrange_continuity true`? >>>> >>> >>> There is a comment in the code about that: >>> >>> /* XXX FIXME Requires DMPlexSetClosurePermutationLexicographic() */ >>> >>> So what is currently done is you project into the discontinuous space >>> from the GMsh coordinates, >>> and then we get the continuous coordinates from those later. This is why >>> we get the right answer. >>> >>> >> Sorry, I'm having difficulty understanding the comment and fully >> understanding your intended meaning. Are you saying that we can only >> project the space to a discontinuous space? >> > > For higher order simplices, because we do not have the mapping to the GMsh > order yet. > > >> Additionally, should we always set >> `dm_plex_gmsh_project_petscdualspace_lagrange_continuity` to false for >> high-order gmsh files? >> > > This is done automatically if you do not override it. > > >> With the option set to `true`, I got the following error: >> > > Yes, do not do that. > > Thanks, > > Matt > > >> ``` >> $ $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/runex33_gmsh_3d_q2.sh >> -e "-dm_plex_gmsh_project_petscdualspace_lagrange_continuity true" >> not ok dm_impls_plex_tests-ex33_gmsh_3d_q2 # Error code: 77 >> # Volume: 0.46875 >> # [0]PETSC ERROR: - Error Message >> -- >> # [0]PETSC ERROR: Petsc has generated inconsistent data >> # [0]PETSC ERROR: Calculated volume 0.46875 != 1. actual volume >> (error 0.53125 > 1e-06 tol) >> # [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [0]PETSC ERROR: Petsc Development GIT revision: >> v3.19.1-294-g9cc24bc9b93 GIT Date: 2023-05-15 12:07:10 + >> # [0]PETSC ERROR: ../ex33 on a arch-linux-c-debug named AMA-PC-RA18 >> by yzz Mon May 15 21:53:43 2023 >> # [0]PETSC ERROR: Configure options >> --CFLAGS=-I/opt/intel/oneapi/mkl/latest/include >> --CXXFLAGS=-I/opt/intel/oneapi/mkl/latest/include >> --LDFLAGS="-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64 >> -L/opt/intel/oneapi/mkl/latest/lib/intel64" --download-bison >> --download-chaco --download-cmake >> --download-eigen="/home/yzz/firedrake/complex-int32-mkl-X-debug/src/eigen-3.3.3.tgz >> " --download-fftw --download-hdf5 --download-hpddm --download-hwloc >> --download-libpng --download-metis --download-mmg --download-mpich >> --download-mumps --download-netcdf --download-p4est --download-parmmg >> --download-pastix --download-pnetcdf --download-ptscotch >> --download-scalapack --download-slepc --download-suitesparse >> --download-superlu_dist --download-tetgen --download-triangle >> --with-blaslapack-dir=/opt/intel/oneapi/mkl/latest --with-c2html=0 >> --with-debugging=1 --with-fortran-bindings=0 >> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/latest >> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/latest >> --with-scalar-type=complex --with-shared-libraries=1 --with-x=1 --with-zlib >> PETSC_ARCH=arch-linux-c-debug >> # [0]PETSC ERROR: #1 CheckVolume() at >> /home/yzz/opt/petsc/src/dm/impls/plex/tests/ex33.c:246 >> # [0]PETSC ERROR: #2 main() at >> /home/yzz/opt/petsc/src/dm/impls/plex/tests/ex33.c:261 >> # [0]PETSC ERROR: PETSc Option Table entries: >> # [0]PETSC ERROR: -coord_space 0 (source: command line) >> # [0]PETSC ERROR: -dm_plex_filename >> /home/yzz/opt/petsc/share/petsc/datafiles/meshes/cube_q2.msh (source: >> command line) >> # [0]PETSC ERROR: -dm_plex_gmsh_project (source: command line) >> # [0]PETSC ERROR: >> -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true (source: >> command line) >> # [0]PETSC ERROR: -tol 1e-6 (source: command line) >> # [0]PETSC ERROR: -volume 1.0 (source: command line) >> # [0]PETSC ERROR: End of Error Message ---send >> entire error message to petsc-ma...@mcs.anl.gov---
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
On Mon, 15 May 2023 at 17:24, Matthew Knepley wrote: > On Sun, May 14, 2023 at 7:23 PM Zongze Yang wrote: > >> Could you try to project the coordinates into the continuity space by >> enabling the option >> `-dm_plex_gmsh_project_petscdualspace_lagrange_continuity true`? >> > > There is a comment in the code about that: > > /* XXX FIXME Requires DMPlexSetClosurePermutationLexicographic() */ > > So what is currently done is you project into the discontinuous space from > the GMsh coordinates, > and then we get the continuous coordinates from those later. This is why > we get the right answer. > > Sorry, I'm having difficulty understanding the comment and fully understanding your intended meaning. Are you saying that we can only project the space to a discontinuous space? Additionally, should we always set `dm_plex_gmsh_project_petscdualspace_lagrange_continuity` to false for high-order gmsh files? With the option set to `true`, I got the following error: ``` $ $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/runex33_gmsh_3d_q2.sh -e "-dm_plex_gmsh_project_petscdualspace_lagrange_continuity true" not ok dm_impls_plex_tests-ex33_gmsh_3d_q2 # Error code: 77 # Volume: 0.46875 # [0]PETSC ERROR: - Error Message -- # [0]PETSC ERROR: Petsc has generated inconsistent data # [0]PETSC ERROR: Calculated volume 0.46875 != 1. actual volume (error 0.53125 > 1e-06 tol) # [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [0]PETSC ERROR: Petsc Development GIT revision: v3.19.1-294-g9cc24bc9b93 GIT Date: 2023-05-15 12:07:10 + # [0]PETSC ERROR: ../ex33 on a arch-linux-c-debug named AMA-PC-RA18 by yzz Mon May 15 21:53:43 2023 # [0]PETSC ERROR: Configure options --CFLAGS=-I/opt/intel/oneapi/mkl/latest/include --CXXFLAGS=-I/opt/intel/oneapi/mkl/latest/include --LDFLAGS="-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64 -L/opt/intel/oneapi/mkl/latest/lib/intel64" --download-bison --download-chaco --download-cmake --download-eigen="/home/yzz/firedrake/complex-int32-mkl-X-debug/src/eigen-3.3.3.tgz " --download-fftw --download-hdf5 --download-hpddm --download-hwloc --download-libpng --download-metis --download-mmg --download-mpich --download-mumps --download-netcdf --download-p4est --download-parmmg --download-pastix --download-pnetcdf --download-ptscotch --download-scalapack --download-slepc --download-suitesparse --download-superlu_dist --download-tetgen --download-triangle --with-blaslapack-dir=/opt/intel/oneapi/mkl/latest --with-c2html=0 --with-debugging=1 --with-fortran-bindings=0 --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/latest --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/latest --with-scalar-type=complex --with-shared-libraries=1 --with-x=1 --with-zlib PETSC_ARCH=arch-linux-c-debug # [0]PETSC ERROR: #1 CheckVolume() at /home/yzz/opt/petsc/src/dm/impls/plex/tests/ex33.c:246 # [0]PETSC ERROR: #2 main() at /home/yzz/opt/petsc/src/dm/impls/plex/tests/ex33.c:261 # [0]PETSC ERROR: PETSc Option Table entries: # [0]PETSC ERROR: -coord_space 0 (source: command line) # [0]PETSC ERROR: -dm_plex_filename /home/yzz/opt/petsc/share/petsc/datafiles/meshes/cube_q2.msh (source: command line) # [0]PETSC ERROR: -dm_plex_gmsh_project (source: command line) # [0]PETSC ERROR: -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true (source: command line) # [0]PETSC ERROR: -tol 1e-6 (source: command line) # [0]PETSC ERROR: -volume 1.0 (source: command line) # [0]PETSC ERROR: End of Error Message ---send entire error message to petsc-ma...@mcs.anl.gov-- # application called MPI_Abort(MPI_COMM_SELF, 77) - process 0 ok dm_impls_plex_tests-ex33_gmsh_3d_q2 # SKIP Command failed so no diff ``` Best wishes, Zongze Thanks, > > Matt > > >> Best wishes, >> Zongze >> >> >> On Mon, 15 May 2023 at 04:24, Matthew Knepley wrote: >> >>> On Sun, May 14, 2023 at 12:27 PM Zongze Yang >>> wrote: >>> >>>> >>>> >>>> >>>> On Sun, 14 May 2023 at 23:54, Matthew Knepley >>>> wrote: >>>> >>>>> On Sun, May 14, 2023 at 9:21 AM Zongze Yang >>>>> wrote: >>>>> >>>>>> Hi, Matt, >>>>>> >>>>>> The issue has been resolved while testing on the latest version of >>>>>> PETSc. It seems that the problem has been fixed in the following merge >>>>>> request: https://gitlab.com/petsc/petsc/-/merge_requests/5970 >>>>>> >>>>> >>>>> No problem. Glad it is working. >>>>>
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Could you try to project the coordinates into the continuity space by enabling the option `-dm_plex_gmsh_project_petscdualspace_lagrange_continuity true`? Best wishes, Zongze On Mon, 15 May 2023 at 04:24, Matthew Knepley wrote: > On Sun, May 14, 2023 at 12:27 PM Zongze Yang wrote: > >> >> >> >> On Sun, 14 May 2023 at 23:54, Matthew Knepley wrote: >> >>> On Sun, May 14, 2023 at 9:21 AM Zongze Yang >>> wrote: >>> >>>> Hi, Matt, >>>> >>>> The issue has been resolved while testing on the latest version of >>>> PETSc. It seems that the problem has been fixed in the following merge >>>> request: https://gitlab.com/petsc/petsc/-/merge_requests/5970 >>>> >>> >>> No problem. Glad it is working. >>> >>> >>>> I sincerely apologize for any inconvenience caused by my previous >>>> message. However, I would like to provide you with additional information >>>> regarding the test files. Attached to this email, you will find two Gmsh >>>> files: "square_2rd.msh" and "square_3rd.msh." These files contain >>>> high-order triangulated mesh data for the unit square. >>>> >>>> ``` >>>> $ ./ex33 -coord_space 0 -dm_plex_filename square_2rd.msh >>>> -dm_plex_gmsh_project >>>> -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true >>>> -dm_plex_gmsh_project_fe_view -volume 1 >>>> PetscFE Object: P2 1 MPI process >>>> type: basic >>>> Basic Finite Element in 2 dimensions with 2 components >>>> PetscSpace Object: P2 1 MPI process >>>> type: sum >>>> Space in 2 variables with 2 components, size 12 >>>> Sum space of 2 concatenated subspaces (all identical) >>>> PetscSpace Object: sum component (sumcomp_) 1 MPI process >>>> type: poly >>>> Space in 2 variables with 1 components, size 6 >>>> Polynomial space of degree 2 >>>> PetscDualSpace Object: P2 1 MPI process >>>> type: lagrange >>>> Dual space with 2 components, size 12 >>>> Continuous Lagrange dual space >>>> Quadrature on a triangle of order 5 on 9 points (dim 2) >>>> Volume: 1. >>>> $ ./ex33 -coord_space 0 -dm_plex_filename square_3rd.msh >>>> -dm_plex_gmsh_project >>>> -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true >>>> -dm_plex_gmsh_project_fe_view -volume 1 >>>> PetscFE Object: P3 1 MPI process >>>> type: basic >>>> Basic Finite Element in 2 dimensions with 2 components >>>> PetscSpace Object: P3 1 MPI process >>>> type: sum >>>> Space in 2 variables with 2 components, size 20 >>>> Sum space of 2 concatenated subspaces (all identical) >>>> PetscSpace Object: sum component (sumcomp_) 1 MPI process >>>> type: poly >>>> Space in 2 variables with 1 components, size 10 >>>> Polynomial space of degree 3 >>>> PetscDualSpace Object: P3 1 MPI process >>>> type: lagrange >>>> Dual space with 2 components, size 20 >>>> Continuous Lagrange dual space >>>> Quadrature on a triangle of order 7 on 16 points (dim 2) >>>> Volume: 1. >>>> ``` >>>> >>>> Thank you for your attention and understanding. I apologize once again >>>> for my previous oversight. >>>> >>> >>> Great! If you make an MR for this, you will be included on the next list >>> of PETSc contributors. Otherwise, I can do it. >>> >>> >> I appreciate your offer to handle the MR. Please go ahead and take care >> of it. Thank you! >> > > I have created the MR with your tests. They are working for me: > > https://gitlab.com/petsc/petsc/-/merge_requests/6463 > > Thanks, > > Matt > > >> Best Wishes, >> Zongze >> >> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best wishes, >>>> Zongze >>>> >>>> >>>> On Sun, 14 May 2023 at 16:44, Matthew Knepley >>>> wrote: >>>> >>>>> On Sat, May 13, 2023 at 6:08 AM Zongze Yang >>>>> wrote: >>>>> >>>>>> Hi, Matt, >>>>>> >>>>>> There seem to be ongoing issues with pr
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
On Sun, 14 May 2023 at 23:54, Matthew Knepley wrote: > On Sun, May 14, 2023 at 9:21 AM Zongze Yang wrote: > >> Hi, Matt, >> >> The issue has been resolved while testing on the latest version of PETSc. >> It seems that the problem has been fixed in the following merge request: >> https://gitlab.com/petsc/petsc/-/merge_requests/5970 >> > > No problem. Glad it is working. > > >> I sincerely apologize for any inconvenience caused by my previous >> message. However, I would like to provide you with additional information >> regarding the test files. Attached to this email, you will find two Gmsh >> files: "square_2rd.msh" and "square_3rd.msh." These files contain >> high-order triangulated mesh data for the unit square. >> >> ``` >> $ ./ex33 -coord_space 0 -dm_plex_filename square_2rd.msh >> -dm_plex_gmsh_project >> -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true >> -dm_plex_gmsh_project_fe_view -volume 1 >> PetscFE Object: P2 1 MPI process >> type: basic >> Basic Finite Element in 2 dimensions with 2 components >> PetscSpace Object: P2 1 MPI process >> type: sum >> Space in 2 variables with 2 components, size 12 >> Sum space of 2 concatenated subspaces (all identical) >> PetscSpace Object: sum component (sumcomp_) 1 MPI process >> type: poly >> Space in 2 variables with 1 components, size 6 >> Polynomial space of degree 2 >> PetscDualSpace Object: P2 1 MPI process >> type: lagrange >> Dual space with 2 components, size 12 >> Continuous Lagrange dual space >> Quadrature on a triangle of order 5 on 9 points (dim 2) >> Volume: 1. >> $ ./ex33 -coord_space 0 -dm_plex_filename square_3rd.msh >> -dm_plex_gmsh_project >> -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true >> -dm_plex_gmsh_project_fe_view -volume 1 >> PetscFE Object: P3 1 MPI process >> type: basic >> Basic Finite Element in 2 dimensions with 2 components >> PetscSpace Object: P3 1 MPI process >> type: sum >> Space in 2 variables with 2 components, size 20 >> Sum space of 2 concatenated subspaces (all identical) >> PetscSpace Object: sum component (sumcomp_) 1 MPI process >> type: poly >> Space in 2 variables with 1 components, size 10 >> Polynomial space of degree 3 >> PetscDualSpace Object: P3 1 MPI process >> type: lagrange >> Dual space with 2 components, size 20 >> Continuous Lagrange dual space >> Quadrature on a triangle of order 7 on 16 points (dim 2) >> Volume: 1. >> ``` >> >> Thank you for your attention and understanding. I apologize once again >> for my previous oversight. >> > > Great! If you make an MR for this, you will be included on the next list > of PETSc contributors. Otherwise, I can do it. > > I appreciate your offer to handle the MR. Please go ahead and take care of it. Thank you! Best Wishes, Zongze > Thanks, > > Matt > > >> Best wishes, >> Zongze >> >> >> On Sun, 14 May 2023 at 16:44, Matthew Knepley wrote: >> >>> On Sat, May 13, 2023 at 6:08 AM Zongze Yang >>> wrote: >>> >>>> Hi, Matt, >>>> >>>> There seem to be ongoing issues with projecting high-order coordinates >>>> from a gmsh file to other spaces. I would like to inquire whether there are >>>> any plans to resolve this problem. >>>> >>>> Thank you for your attention to this matter. >>>> >>> >>> Yes, I will look at it. The important thing is to have a good test. Here >>> are the higher order geometry tests >>> >>> >>> https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tests/ex33.c >>> >>> I take shapes with known volume, mesh them with higher order geometry, >>> and look at the convergence to the true volume. Could you add a GMsh test, >>> meaning the .msh file and known volume, and I will fix it? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> >>>> Best wishes, >>>> Zongze >>>> >>>> >>>> On Sat, 18 Jun 2022 at 20:31, Zongze Yang wrote: >>>> >>>>> Thank you for your reply. May I ask for some references on the order >>>>> of the dofs on PETSc's FE Space (especially high order elements)? >>>>> >>>>> Thanks, >>>>> &
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Yes, you are correct. I have conducted tests using high-order 3D meshes of a unit cube, and regrettably, the tests have failed. I have attached the files for your reference. Kindly review the output provided below: ( The volume should be 1) ``` $ mpiexec -n 3 ./ex33 -coord_space 0 -dm_plex_filename cube_2rd.msh -dm_plex_gmsh_project -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true -dm_plex_gmsh_project_fe_view PetscFE Object: P2 3 MPI processes type: basic Basic Finite Element in 3 dimensions with 3 components PetscSpace Object: P2 3 MPI processes type: sum Space in 3 variables with 3 components, size 30 Sum space of 3 concatenated subspaces (all identical) PetscSpace Object: sum component (sumcomp_) 3 MPI processes type: poly Space in 3 variables with 1 components, size 10 Polynomial space of degree 2 PetscDualSpace Object: P2 3 MPI processes type: lagrange Dual space with 3 components, size 30 Continuous Lagrange dual space Quadrature on a tetrahedron of order 5 on 27 points (dim 3) Volume: 0.46875 $ mpiexec -n 3 ./ex33 -coord_space 0 -dm_plex_filename cube_3rd.msh -dm_plex_gmsh_project -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true -dm_plex_gmsh_project_fe_view PetscFE Object: P3 3 MPI processes type: basic Basic Finite Element in 3 dimensions with 3 components PetscSpace Object: P3 3 MPI processes type: sum Space in 3 variables with 3 components, size 60 Sum space of 3 concatenated subspaces (all identical) PetscSpace Object: sum component (sumcomp_) 3 MPI processes type: poly Space in 3 variables with 1 components, size 20 Polynomial space of degree 3 PetscDualSpace Object: P3 3 MPI processes type: lagrange Dual space with 3 components, size 60 Continuous Lagrange dual space Quadrature on a tetrahedron of order 7 on 64 points (dim 3) Volume: 0.536855 ``` Best wishes, Zongze On Sun, 14 May 2023 at 22:36, Jed Brown wrote: > Good to hear this works for you. I believe there is still a problem with > high order tetrahedral elements (we've been coping with it for months and > someone asked last week) and plan to look at it as soon as possible now > that my semester finished. > > Zongze Yang writes: > > > Hi, Matt, > > > > The issue has been resolved while testing on the latest version of PETSc. > > It seems that the problem has been fixed in the following merge request: > > https://gitlab.com/petsc/petsc/-/merge_requests/5970 > > > > I sincerely apologize for any inconvenience caused by my previous > message. > > However, I would like to provide you with additional information > regarding > > the test files. Attached to this email, you will find two Gmsh files: > > "square_2rd.msh" and "square_3rd.msh." These files contain high-order > > triangulated mesh data for the unit square. > > > > ``` > > $ ./ex33 -coord_space 0 -dm_plex_filename square_2rd.msh > > -dm_plex_gmsh_project > > -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true > > -dm_plex_gmsh_project_fe_view -volume 1 > > PetscFE Object: P2 1 MPI process > > type: basic > > Basic Finite Element in 2 dimensions with 2 components > > PetscSpace Object: P2 1 MPI process > > type: sum > > Space in 2 variables with 2 components, size 12 > > Sum space of 2 concatenated subspaces (all identical) > > PetscSpace Object: sum component (sumcomp_) 1 MPI process > > type: poly > > Space in 2 variables with 1 components, size 6 > > Polynomial space of degree 2 > > PetscDualSpace Object: P2 1 MPI process > > type: lagrange > > Dual space with 2 components, size 12 > > Continuous Lagrange dual space > > Quadrature on a triangle of order 5 on 9 points (dim 2) > > Volume: 1. > > $ ./ex33 -coord_space 0 -dm_plex_filename square_3rd.msh > > -dm_plex_gmsh_project > > -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true > > -dm_plex_gmsh_project_fe_view -volume 1 > > PetscFE Object: P3 1 MPI process > > type: basic > > Basic Finite Element in 2 dimensions with 2 components > > PetscSpace Object: P3 1 MPI process > > type: sum > > Space in 2 variables with 2 components, size 20 > > Sum space of 2 concatenated subspaces (all identical) > > PetscSpace Object: sum component (sumcomp_) 1 MPI process > > type: poly > > Space in 2 variables with 1 components, size 10 > > Polynomial space of degree 3 > > PetscDualSpace Object: P3 1 MPI process > > type: lagrange > > Dual space with 2 components, size 20 > > C
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Hi, Matt, The issue has been resolved while testing on the latest version of PETSc. It seems that the problem has been fixed in the following merge request: https://gitlab.com/petsc/petsc/-/merge_requests/5970 I sincerely apologize for any inconvenience caused by my previous message. However, I would like to provide you with additional information regarding the test files. Attached to this email, you will find two Gmsh files: "square_2rd.msh" and "square_3rd.msh." These files contain high-order triangulated mesh data for the unit square. ``` $ ./ex33 -coord_space 0 -dm_plex_filename square_2rd.msh -dm_plex_gmsh_project -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true -dm_plex_gmsh_project_fe_view -volume 1 PetscFE Object: P2 1 MPI process type: basic Basic Finite Element in 2 dimensions with 2 components PetscSpace Object: P2 1 MPI process type: sum Space in 2 variables with 2 components, size 12 Sum space of 2 concatenated subspaces (all identical) PetscSpace Object: sum component (sumcomp_) 1 MPI process type: poly Space in 2 variables with 1 components, size 6 Polynomial space of degree 2 PetscDualSpace Object: P2 1 MPI process type: lagrange Dual space with 2 components, size 12 Continuous Lagrange dual space Quadrature on a triangle of order 5 on 9 points (dim 2) Volume: 1. $ ./ex33 -coord_space 0 -dm_plex_filename square_3rd.msh -dm_plex_gmsh_project -dm_plex_gmsh_project_petscdualspace_lagrange_continuity true -dm_plex_gmsh_project_fe_view -volume 1 PetscFE Object: P3 1 MPI process type: basic Basic Finite Element in 2 dimensions with 2 components PetscSpace Object: P3 1 MPI process type: sum Space in 2 variables with 2 components, size 20 Sum space of 2 concatenated subspaces (all identical) PetscSpace Object: sum component (sumcomp_) 1 MPI process type: poly Space in 2 variables with 1 components, size 10 Polynomial space of degree 3 PetscDualSpace Object: P3 1 MPI process type: lagrange Dual space with 2 components, size 20 Continuous Lagrange dual space Quadrature on a triangle of order 7 on 16 points (dim 2) Volume: 1. ``` Thank you for your attention and understanding. I apologize once again for my previous oversight. Best wishes, Zongze On Sun, 14 May 2023 at 16:44, Matthew Knepley wrote: > On Sat, May 13, 2023 at 6:08 AM Zongze Yang wrote: > >> Hi, Matt, >> >> There seem to be ongoing issues with projecting high-order coordinates >> from a gmsh file to other spaces. I would like to inquire whether there are >> any plans to resolve this problem. >> >> Thank you for your attention to this matter. >> > > Yes, I will look at it. The important thing is to have a good test. Here > are the higher order geometry tests > > > https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tests/ex33.c > > I take shapes with known volume, mesh them with higher order geometry, and > look at the convergence to the true volume. Could you add a GMsh test, > meaning the .msh file and known volume, and I will fix it? > > Thanks, > > Matt > > >> Best wishes, >> Zongze >> >> >> On Sat, 18 Jun 2022 at 20:31, Zongze Yang wrote: >> >>> Thank you for your reply. May I ask for some references on the order of >>> the dofs on PETSc's FE Space (especially high order elements)? >>> >>> Thanks, >>> >>> Zongze >>> >>> Matthew Knepley 于2022年6月18日周六 20:02写道： >>> >>>> On Sat, Jun 18, 2022 at 2:16 AM Zongze Yang >>>> wrote: >>>> >>>>> In order to check if I made mistakes in the python code, I try to use >>>>> c code to show the issue on DMProjectCoordinates. The code and mesh file >>>>> is >>>>> attached. >>>>> If the code is correct, there must be something wrong with >>>>> `DMProjectCoordinates` or `DMPlexCreateGmshFromFile` for high-order mesh. >>>>> >>>> >>>> Something is definitely wrong with high order, periodic simplices from >>>> Gmsh. We had not tested that case. I am at a conference and cannot look at >>>> it for a week. >>>> My suspicion is that the space we make when reading in the Gmsh >>>> coordinates does not match the values (wrong order). >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> The command and the output are listed below: (Obviously the bounding >>>>> box is changed.) >>>>> ``` >>>>> $ ./test_gmsh_load_2rd -filename cube-p2.msh -old_fe_view -new_fe_view >&
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Hi, Matt, There seem to be ongoing issues with projecting high-order coordinates from a gmsh file to other spaces. I would like to inquire whether there are any plans to resolve this problem. Thank you for your attention to this matter. Best wishes, Zongze On Sat, 18 Jun 2022 at 20:31, Zongze Yang wrote: > Thank you for your reply. May I ask for some references on the order of > the dofs on PETSc's FE Space (especially high order elements)? > > Thanks, > > Zongze > > Matthew Knepley 于2022年6月18日周六 20:02写道： > >> On Sat, Jun 18, 2022 at 2:16 AM Zongze Yang wrote: >> >>> In order to check if I made mistakes in the python code, I try to use c >>> code to show the issue on DMProjectCoordinates. The code and mesh file is >>> attached. >>> If the code is correct, there must be something wrong with >>> `DMProjectCoordinates` or `DMPlexCreateGmshFromFile` for high-order mesh. >>> >> >> Something is definitely wrong with high order, periodic simplices from >> Gmsh. We had not tested that case. I am at a conference and cannot look at >> it for a week. >> My suspicion is that the space we make when reading in the Gmsh >> coordinates does not match the values (wrong order). >> >> Thanks, >> >> Matt >> >> >>> The command and the output are listed below: (Obviously the bounding box >>> is changed.) >>> ``` >>> $ ./test_gmsh_load_2rd -filename cube-p2.msh -old_fe_view -new_fe_view >>> Old Bounding Box: >>> 0: lo = 0. hi = 1. >>> 1: lo = 0. hi = 1. >>> 2: lo = 0. hi = 1. >>> PetscFE Object: OldCoordinatesFE 1 MPI processes >>> type: basic >>> Basic Finite Element in 3 dimensions with 3 components >>> PetscSpace Object: P2 1 MPI processes >>> type: sum >>> Space in 3 variables with 3 components, size 30 >>> Sum space of 3 concatenated subspaces (all identical) >>> PetscSpace Object: sum component (sumcomp_) 1 MPI processes >>> type: poly >>> Space in 3 variables with 1 components, size 10 >>> Polynomial space of degree 2 >>> PetscDualSpace Object: P2 1 MPI processes >>> type: lagrange >>> Dual space with 3 components, size 30 >>> Discontinuous Lagrange dual space >>> Quadrature of order 5 on 27 points (dim 3) >>> PetscFE Object: NewCoordinatesFE 1 MPI processes >>> type: basic >>> Basic Finite Element in 3 dimensions with 3 components >>> PetscSpace Object: P2 1 MPI processes >>> type: sum >>> Space in 3 variables with 3 components, size 30 >>> Sum space of 3 concatenated subspaces (all identical) >>> PetscSpace Object: sum component (sumcomp_) 1 MPI processes >>> type: poly >>> Space in 3 variables with 1 components, size 10 >>> Polynomial space of degree 2 >>> PetscDualSpace Object: P2 1 MPI processes >>> type: lagrange >>> Dual space with 3 components, size 30 >>> Continuous Lagrange dual space >>> Quadrature of order 5 on 27 points (dim 3) >>> New Bounding Box: >>> 0: lo = 2.5624e-17 hi = 8. >>> 1: lo = -9.23372e-17 hi = 7. >>> 2: lo = 2.72091e-17 hi = 8.5 >>> ``` >>> >>> Thanks, >>> Zongze >>> >>> Zongze Yang 于2022年6月17日周五 14:54写道： >>> >>>> I tried the projection operation. However, it seems that the projection >>>> gives the wrong solution. After projection, the bounding box is changed! >>>> See logs below. >>>> >>>> First, I patch the petsc4py by adding `DMProjectCoordinates`: >>>> ``` >>>> diff --git a/src/binding/petsc4py/src/PETSc/DM.pyx >>>> b/src/binding/petsc4py/src/PETSc/DM.pyx >>>> index d8a58d183a..dbcdb280f1 100644 >>>> --- a/src/binding/petsc4py/src/PETSc/DM.pyx >>>> +++ b/src/binding/petsc4py/src/PETSc/DM.pyx >>>> @@ -307,6 +307,12 @@ cdef class DM(Object): >>>> PetscINCREF(c.obj) >>>> return c >>>> >>>> +def projectCoordinates(self, FE fe=None): >>>> +if fe is None: >>>> +CHKERR( DMProjectCoordinates(self.dm, NULL) ) >>>> +else: >>>> +CHKERR( DMProjectCoordinates(self.dm, fe.fe) ) >>>> + >>>> def getBoundingBox(self): >>>> cdef PetscInt i,dim=0 >>>> CHKERR( DMGetCo
Re: [petsc-users] Build error: vecimpl.h:124:98: error: expected declaration specifiers or '...' before string constant
The value of PETSC_C_VERSION = 17. I changed `PETSC_C_VERSION >= 17`
to `PETSC_C_VERSION
> 17`, then the error is gone.
Best wishes,
Zongze
On Tue, 18 Apr 2023 at 22:36, Satish Balay wrote:
> I think its best if configure can handle this automatically (check for
> broken compilers). Until then - perhaps we should use:
>
>
> diff --git a/include/petsc/private/vecimpl.h
> b/include/petsc/private/vecimpl.h
> index dd75dbbc00b..dd9ef6791c5 100644
> --- a/include/petsc/private/vecimpl.h
> +++ b/include/petsc/private/vecimpl.h
> @@ -110,12 +110,7 @@ struct _VecOps {
>PetscErrorCode (*setvaluescoo)(Vec, const PetscScalar[], InsertMode);
> };
>
> -#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >= 11))
> - #if (PETSC_C_VERSION >= 11) && (PETSC_C_VERSION < 23)
> -// static_assert() is a keyword since C23, before that defined as
> macro in assert.h
> -#include
> - #endif
> -
> +#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >= 17))
> static_assert(offsetof(struct _VecOps, duplicate) == sizeof(void
> (*)(void)) * VECOP_DUPLICATE, "");
> static_assert(offsetof(struct _VecOps, set) == sizeof(void (*)(void)) *
> VECOP_SET, "");
> static_assert(offsetof(struct _VecOps, view) == sizeof(void (*)(void)) *
> VECOP_VIEW, "");
>
>
> Or just:
>
> +#if defined(offsetof) && defined(__cplusplus)
>
> Satish
>
> On Tue, 18 Apr 2023, Jacob Faibussowitsch wrote:
>
> > This is a bug in GCC 9. Can you try the following:
> >
> > $ make clean
> > $ make CFLAGS+='-std=gnu11’
> >
> > Best regards,
> >
> > Jacob Faibussowitsch
> > (Jacob Fai - booss - oh - vitch)
> >
> > > On Apr 18, 2023, at 10:07, Zongze Yang wrote:
> > >
> > > No, it doesn't. It has the same problem. I just `make clean` and the
> `make`. Do I need to reconfigure?
> > >
> > > Best wishes,
> > > Zongze
> > >
> > >
> > > On Tue, 18 Apr 2023 at 21:09, Satish Balay wrote:
> > > Does this change work?
> > >
> > > diff --git a/include/petsc/private/vecimpl.h
> b/include/petsc/private/vecimpl.h
> > > index dd75dbbc00b..168540b546e 100644
> > > --- a/include/petsc/private/vecimpl.h
> > > +++ b/include/petsc/private/vecimpl.h
> > > @@ -110,7 +110,7 @@ struct _VecOps {
> > >PetscErrorCode (*setvaluescoo)(Vec, const PetscScalar[],
> InsertMode);
> > > };
> > >
> > > -#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >=
> 11))
> > > +#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >=
> 17))
> > >#if (PETSC_C_VERSION >= 11) && (PETSC_C_VERSION < 23)
> > > // static_assert() is a keyword since C23, before that defined as
> macro in assert.h
> > > #include
> > >
> > >
> > > Satish
> > >
> > > On Tue, 18 Apr 2023, Zongze Yang wrote:
> > >
> > > > Hi, I am building petsc using gcc@9.5.0, and found the following
> error:
> > > >
> > > > ```
> > > > In file included from /usr/include/alloca.h:25,
> > > > from /usr/include/stdlib.h:497,
> > > > from
> > > >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petscsys.h:1395,
> > > > from
> > > > /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petscsf.h:7,
> > > > from
> > > >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/src/vec/is/sf/interface/vscat.c:1:
> > > >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petsc/private/vecimpl.h:124:15:
> > > > error: expected declaration specifiers or '...' before
> '__builtin_offsetof'
> > > > 124 | static_assert(offsetof(struct _VecOps, loadnative) ==
> sizeof(void
> > > > (*)(void)) * VECOP_LOADNATIVE, "");
> > > > | ^~~~
> > > > In file included from
> > > >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/src/vec/is/sf/interface/vscat.c:7:
> > > >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petsc/private/vecimpl.h:124:98:
> > > > error: expected declaration specifiers or '...' before string
> constant
> > > > 124 | static_assert(offsetof(struct _VecOps, loadnative) ==
> sizeof(void
> > > > (*)(void)) * VECOP_LOADNATIVE, "");
> > > > |
> > > > ^~
> > > > ```
> > > >
> > > > Could someone give me some hints to fix it? The configure.log and
> make.log
> > > > are attached.
> > > >
> > > >
> > > > Best wishes,
> > > > Zongze
> > > >
> > >
> >
>
Re: [petsc-users] Build error: vecimpl.h:124:98: error: expected declaration specifiers or '...' before string constant
No, it doesn't. It has the same problem. I just `make clean` and the
`make`. Do I need to reconfigure?
Best wishes,
Zongze
On Tue, 18 Apr 2023 at 21:09, Satish Balay wrote:
> Does this change work?
>
> diff --git a/include/petsc/private/vecimpl.h
> b/include/petsc/private/vecimpl.h
> index dd75dbbc00b..168540b546e 100644
> --- a/include/petsc/private/vecimpl.h
> +++ b/include/petsc/private/vecimpl.h
> @@ -110,7 +110,7 @@ struct _VecOps {
>PetscErrorCode (*setvaluescoo)(Vec, const PetscScalar[], InsertMode);
> };
>
> -#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >= 11))
> +#if defined(offsetof) && (defined(__cplusplus) || (PETSC_C_VERSION >= 17))
>#if (PETSC_C_VERSION >= 11) && (PETSC_C_VERSION < 23)
> // static_assert() is a keyword since C23, before that defined as
> macro in assert.h
> #include
>
>
> Satish
>
> On Tue, 18 Apr 2023, Zongze Yang wrote:
>
> > Hi, I am building petsc using gcc@9.5.0, and found the following error:
> >
> > ```
> > In file included from /usr/include/alloca.h:25,
> > from /usr/include/stdlib.h:497,
> > from
> > /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petscsys.h:1395,
> > from
> > /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petscsf.h:7,
> > from
> >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/src/vec/is/sf/interface/vscat.c:1:
> >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petsc/private/vecimpl.h:124:15:
> > error: expected declaration specifiers or '...' before
> '__builtin_offsetof'
> > 124 | static_assert(offsetof(struct _VecOps, loadnative) == sizeof(void
> > (*)(void)) * VECOP_LOADNATIVE, "");
> > | ^~~~
> > In file included from
> >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/src/vec/is/sf/interface/vscat.c:7:
> >
> /home/lrtfm/opt/firedrake/complex-int32/petsc/include/petsc/private/vecimpl.h:124:98:
> > error: expected declaration specifiers or '...' before string constant
> > 124 | static_assert(offsetof(struct _VecOps, loadnative) == sizeof(void
> > (*)(void)) * VECOP_LOADNATIVE, "");
> > |
> > ^~
> > ```
> >
> > Could someone give me some hints to fix it? The configure.log and
> make.log
> > are attached.
> >
> >
> > Best wishes,
> > Zongze
> >
>
>
Re: [petsc-users] `snes+ksponly` did not update the solution when ksp failed.
Thank you for your clarification. Best wishes, Zongze On Mon, 20 Mar 2023 at 20:00, Matthew Knepley wrote: > On Mon, Mar 20, 2023 at 2:41 AM Zongze Yang wrote: > >> Hi, >> >> Hope this email finds you well. I am using firedrake to solve linear >> problems, which use SNES with KSPONLY. >> >> I found that the solution did not update when the `ksp` failed with >> DIVERGED_ITS. >> The macro `SNESCheckKSPSolve` called in `SNESSolve_KSPONLY` make it >> return before the solution is updated. >> > > Yes, this is the intended behavior. We do not guarantee cleanup on errors. > > >> Is this behavior as expected? Can I just increase the value of >> `maxLinearSolveFailures` >> to make the solution updated without introducing other side effects? >> > > Yes, that is right. It will not have other side effects with this SNES > type. > > Thanks, > > Matt > > >> Best wishes, >> Zongze >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
[petsc-users] `snes+ksponly` did not update the solution when ksp failed.
Hi, Hope this email finds you well. I am using firedrake to solve linear problems, which use SNES with KSPONLY. I found that the solution did not update when the `ksp` failed with DIVERGED_ITS. The macro `SNESCheckKSPSolve` called in `SNESSolve_KSPONLY` make it return before the solution is updated. Is this behavior as expected? Can I just increase the value of `maxLinearSolveFailures` to make the solution updated without introducing other side effects? Best wishes, Zongze
Re: [petsc-users] petsc4py did not raise for a second time with option `ksp_error_if_not_converged`
Thank you for your suggestion.
Best wishes,
Zongze
On Mon, 6 Mar 2023 at 02:40, Matthew Knepley wrote:
> On Sun, Mar 5, 2023 at 3:14 AM Zongze Yang wrote:
>
>>
>>
>> Hello,
>>
>> I am trying to catch the "not converged" error in a loop with the
>> `ksp_error_if_not_converged` option on. However, it seems that PETSc only
>> raises the exception once, even though the solver does not converge after
>> that. Is this expected behavior? Can I make it raise an exception every
>> time?
>>
>
> When an error is raised, we do not guarantee a consistent state for
> recovery, so errors terminate the program. If you want
> to do something useful with non-convergence, then you do not set
> -ksp_error_if_not_converged. Rather you check the convergence
> code, and if it is not convergence, you take your action.
>
> Thanks,
>
> Matt
>
>
>> I have included a code snippet of the loop below, and the complete code
>> is attached:
>> ```python
>> for i in range(3):
>> printf(f"Loop i = {i}")
>> try:
>> solver.solve()
>> except ConvergenceError:
>> printf(f" Error from Firedrake: solver did not converged:
>> {get_ksp_reason(solver)}")
>> except PETSc.Error as e:
>> if e.ierr == 91:
>> printf(f" Error from PETSc: solver did not converged:
>> {get_ksp_reason(solver)}")
>> else:
>> raise
>> ```
>>
>> The output of the code looks like this:
>> ```python
>> (complex-int32-mkl) $ python test_error.py
>> Loop i = 0
>> Linear solve did not converge due to DIVERGED_ITS iterations 4
>> Error from PETSc: solver did not converged: DIVERGED_MAX_IT
>> Loop i = 1
>> Linear solve did not converge due to DIVERGED_ITS iterations 4
>> Error from Firedrake: solver did not converged: DIVERGED_MAX_IT
>> Loop i = 2
>> Linear solve did not converge due to DIVERGED_ITS iterations 4
>> Error from Firedrake: solver did not converged: DIVERGED_MAX_IT
>> ```
>>
>> Best wishes,
>> Zongze
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
[petsc-users] petsc4py did not raise for a second time with option `ksp_error_if_not_converged`
Hello,
I am trying to catch the "not converged" error in a loop with the
`ksp_error_if_not_converged` option on. However, it seems that PETSc only
raises the exception once, even though the solver does not converge after
that. Is this expected behavior? Can I make it raise an exception every
time?
I have included a code snippet of the loop below, and the complete code is
attached:
```python
for i in range(3):
printf(f"Loop i = {i}")
try:
solver.solve()
except ConvergenceError:
printf(f" Error from Firedrake: solver did not converged:
{get_ksp_reason(solver)}")
except PETSc.Error as e:
if e.ierr == 91:
printf(f" Error from PETSc: solver did not converged:
{get_ksp_reason(solver)}")
else:
raise
```
The output of the code looks like this:
```python
(complex-int32-mkl) $ python test_error.py
Loop i = 0
Linear solve did not converge due to DIVERGED_ITS iterations 4
Error from PETSc: solver did not converged: DIVERGED_MAX_IT
Loop i = 1
Linear solve did not converge due to DIVERGED_ITS iterations 4
Error from Firedrake: solver did not converged: DIVERGED_MAX_IT
Loop i = 2
Linear solve did not converge due to DIVERGED_ITS iterations 4
Error from Firedrake: solver did not converged: DIVERGED_MAX_IT
```
Best wishes,
Zongze
test_error.py
Description: Binary data
Re: [petsc-users] Random Error of mumps: out of memory: INFOG(1)=-9
Thanks, I will give it a try. Best wishes, Zongze On Sat, 4 Mar 2023 at 23:09, Pierre Jolivet wrote: > > > On 4 Mar 2023, at 3:26 PM, Zongze Yang wrote: > > > > > On Sat, 4 Mar 2023 at 22:03, Pierre Jolivet wrote: > >> >> >> On 4 Mar 2023, at 2:51 PM, Zongze Yang wrote: >> >> >> >> On Sat, 4 Mar 2023 at 21:37, Pierre Jolivet wrote: >> >>> >>> >>> > On 4 Mar 2023, at 2:30 PM, Zongze Yang wrote: >>> > >>> > Hi, >>> > >>> > I am writing to seek your advice regarding a problem I encountered >>> while using multigrid to solve a certain issue. >>> > I am currently using multigrid with the coarse problem solved by PCLU. >>> However, the PC failed randomly with the error below (the value of INFO(2) >>> may differ): >>> > ```shell >>> > [ 0] Error reported by MUMPS in numerical factorization phase: >>> INFOG(1)=-9, INFO(2)=36 >>> > ``` >>> > >>> > Upon checking the documentation of MUMPS, I discovered that increasing >>> the value of ICNTL(14) may help resolve the issue. Specifically, I set the >>> option -mat_mumps_icntl_14 to a higher value (such as 40), and the error >>> seemed to disappear after I set the value of ICNTL(14) to 80. However, I am >>> still curious as to why MUMPS failed randomly in the first place. >>> > >>> > Upon further inspection, I found that the number of nonzeros of the >>> PETSc matrix and the MUMPS matrix were different every time I ran the code. >>> I am now left with the following questions: >>> > >>> > 1. What could be causing the number of nonzeros of the MUMPS matrix to >>> change every time I run the code? >>> >>> Is the Mat being fed to MUMPS distributed on a communicator of size >>> greater than one? >>> If yes, then, depending on the pivoting and the renumbering, you may get >>> non-deterministic results. >>> >> >> Hi, Pierre, >> Thank you for your prompt reply. Yes, the size of the communicator is >> greater than one. >> Even if the size of the communicator is equal, are the results >> still non-deterministic? >> >> >> In the most general case, yes. >> >> Can I assume the Mat being fed to MUMPS is the same in this case? >> >> >> Are you doing algebraic or geometric multigrid? >> Are the prolongation operators computed by Firedrake or by PETSc, e.g., >> through GAMG? >> If it’s the latter, I believe the Mat being fed to MUMPS should always be >> the same. >> If it’s the former, you’ll have to ask the Firedrake people if there may >> be non-determinism in the coarsening process. >> > > I am using geometric multigrid, and the prolongation operators, I think, > are computed by Firedrake. > Thanks for your suggestion, I will ask the Firedrake people. > > >> >> Is the pivoting and renumbering all done by MUMPS other than PETSc? >> >> >> You could provide your own numbering, but by default, this is outsourced >> to MUMPS indeed, which will itself outsourced this to METIS, AMD, etc. >> > > I think I won't do this. > By the way, does the result of superlu_dist have a similar > non-deterministic? > > > SuperLU_DIST uses static pivoting as far as I know, so it may be more > deterministic. > > Thanks, > Pierre > > Thanks, > Zongze > > >> Thanks, >> Pierre >> >> >>> > 2. Why is the number of nonzeros of the MUMPS matrix significantly >>> greater than that of the PETSc matrix (as seen in the output of ksp_view, >>> 115025949 vs 7346177)? >>> >>> Exact factorizations introduce fill-in. >>> The number of nonzeros you are seeing for MUMPS is the number of >>> nonzeros in the factors. >>> >>> > 3. Is it possible that the varying number of nonzeros of the MUMPS >>> matrix is the cause of the random failure? >>> >>> Yes, MUMPS uses dynamic scheduling, which will depend on numerical >>> pivoting, and which may generate factors with different number of nonzeros. >>> >> >> Got it. Thank you for your clear explanation. >> Zongze >> >> >>> Thanks, >>> Pierre >> >> >>> > I have attached a test example written in Firedrake. The output of >>> `ksp_view` after running the code twice is included below for your >>> reference. >>> > In the output, the number of nonzeros of the MUMPS matrix was
Re: [petsc-users] Random Error of mumps: out of memory: INFOG(1)=-9
On Sat, 4 Mar 2023 at 22:03, Pierre Jolivet wrote: > > > On 4 Mar 2023, at 2:51 PM, Zongze Yang wrote: > > > > On Sat, 4 Mar 2023 at 21:37, Pierre Jolivet wrote: > >> >> >> > On 4 Mar 2023, at 2:30 PM, Zongze Yang wrote: >> > >> > Hi, >> > >> > I am writing to seek your advice regarding a problem I encountered >> while using multigrid to solve a certain issue. >> > I am currently using multigrid with the coarse problem solved by PCLU. >> However, the PC failed randomly with the error below (the value of INFO(2) >> may differ): >> > ```shell >> > [ 0] Error reported by MUMPS in numerical factorization phase: >> INFOG(1)=-9, INFO(2)=36 >> > ``` >> > >> > Upon checking the documentation of MUMPS, I discovered that increasing >> the value of ICNTL(14) may help resolve the issue. Specifically, I set the >> option -mat_mumps_icntl_14 to a higher value (such as 40), and the error >> seemed to disappear after I set the value of ICNTL(14) to 80. However, I am >> still curious as to why MUMPS failed randomly in the first place. >> > >> > Upon further inspection, I found that the number of nonzeros of the >> PETSc matrix and the MUMPS matrix were different every time I ran the code. >> I am now left with the following questions: >> > >> > 1. What could be causing the number of nonzeros of the MUMPS matrix to >> change every time I run the code? >> >> Is the Mat being fed to MUMPS distributed on a communicator of size >> greater than one? >> If yes, then, depending on the pivoting and the renumbering, you may get >> non-deterministic results. >> > > Hi, Pierre, > Thank you for your prompt reply. Yes, the size of the communicator is > greater than one. > Even if the size of the communicator is equal, are the results > still non-deterministic? > > > In the most general case, yes. > > Can I assume the Mat being fed to MUMPS is the same in this case? > > > Are you doing algebraic or geometric multigrid? > Are the prolongation operators computed by Firedrake or by PETSc, e.g., > through GAMG? > If it’s the latter, I believe the Mat being fed to MUMPS should always be > the same. > If it’s the former, you’ll have to ask the Firedrake people if there may > be non-determinism in the coarsening process. > I am using geometric multigrid, and the prolongation operators, I think, are computed by Firedrake. Thanks for your suggestion, I will ask the Firedrake people. > > Is the pivoting and renumbering all done by MUMPS other than PETSc? > > > You could provide your own numbering, but by default, this is outsourced > to MUMPS indeed, which will itself outsourced this to METIS, AMD, etc. > I think I won't do this. By the way, does the result of superlu_dist have a similar non-deterministic? Thanks, Zongze > Thanks, > Pierre > > >> > 2. Why is the number of nonzeros of the MUMPS matrix significantly >> greater than that of the PETSc matrix (as seen in the output of ksp_view, >> 115025949 vs 7346177)? >> >> Exact factorizations introduce fill-in. >> The number of nonzeros you are seeing for MUMPS is the number of nonzeros >> in the factors. >> >> > 3. Is it possible that the varying number of nonzeros of the MUMPS >> matrix is the cause of the random failure? >> >> Yes, MUMPS uses dynamic scheduling, which will depend on numerical >> pivoting, and which may generate factors with different number of nonzeros. >> > > Got it. Thank you for your clear explanation. > Zongze > > >> Thanks, >> Pierre > > >> > I have attached a test example written in Firedrake. The output of >> `ksp_view` after running the code twice is included below for your >> reference. >> > In the output, the number of nonzeros of the MUMPS matrix was 115025949 >> and 115377847, respectively, while that of the PETSc matrix was only >> 7346177. >> > >> > ```shell >> > (complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view >> ::ascii_info_detail | grep -A3 "type: " >> > type: preonly >> > maximum iterations=1, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=1. >> > left preconditioning >> > -- >> > type: lu >> > out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: external >> > -- >> > type: mumps >> > rows=1050625, cols=1050625 >> >
Re: [petsc-users] Random Error of mumps: out of memory: INFOG(1)=-9
On Sat, 4 Mar 2023 at 21:37, Pierre Jolivet wrote: > > > > On 4 Mar 2023, at 2:30 PM, Zongze Yang wrote: > > > > Hi, > > > > I am writing to seek your advice regarding a problem I encountered while > using multigrid to solve a certain issue. > > I am currently using multigrid with the coarse problem solved by PCLU. > However, the PC failed randomly with the error below (the value of INFO(2) > may differ): > > ```shell > > [ 0] Error reported by MUMPS in numerical factorization phase: > INFOG(1)=-9, INFO(2)=36 > > ``` > > > > Upon checking the documentation of MUMPS, I discovered that increasing > the value of ICNTL(14) may help resolve the issue. Specifically, I set the > option -mat_mumps_icntl_14 to a higher value (such as 40), and the error > seemed to disappear after I set the value of ICNTL(14) to 80. However, I am > still curious as to why MUMPS failed randomly in the first place. > > > > Upon further inspection, I found that the number of nonzeros of the > PETSc matrix and the MUMPS matrix were different every time I ran the code. > I am now left with the following questions: > > > > 1. What could be causing the number of nonzeros of the MUMPS matrix to > change every time I run the code? > > Is the Mat being fed to MUMPS distributed on a communicator of size > greater than one? > If yes, then, depending on the pivoting and the renumbering, you may get > non-deterministic results. > Hi, Pierre, Thank you for your prompt reply. Yes, the size of the communicator is greater than one. Even if the size of the communicator is equal, are the results still non-deterministic? Can I assume the Mat being fed to MUMPS is the same in this case? Is the pivoting and renumbering all done by MUMPS other than PETSc? > > 2. Why is the number of nonzeros of the MUMPS matrix significantly > greater than that of the PETSc matrix (as seen in the output of ksp_view, > 115025949 vs 7346177)? > > Exact factorizations introduce fill-in. > The number of nonzeros you are seeing for MUMPS is the number of nonzeros > in the factors. > > > 3. Is it possible that the varying number of nonzeros of the MUMPS > matrix is the cause of the random failure? > > Yes, MUMPS uses dynamic scheduling, which will depend on numerical > pivoting, and which may generate factors with different number of nonzeros. > Got it. Thank you for your clear explanation. Zongze > Thanks, > Pierre > > I have attached a test example written in Firedrake. The output of > `ksp_view` after running the code twice is included below for your > reference. > > In the output, the number of nonzeros of the MUMPS matrix was 115025949 > and 115377847, respectively, while that of the PETSc matrix was only > 7346177. > > > > ```shell > > (complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view > ::ascii_info_detail | grep -A3 "type: " > > type: preonly > > maximum iterations=1, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=1. > > left preconditioning > > -- > > type: lu > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: external > > -- > > type: mumps > > rows=1050625, cols=1050625 > > package used to perform factorization: mumps > > total: nonzeros=115025949, allocated nonzeros=115025949 > > -- > > type: mpiaij > > rows=1050625, cols=1050625 > > total: nonzeros=7346177, allocated nonzeros=7346177 > > total number of mallocs used during MatSetValues calls=0 > > (complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view > ::ascii_info_detail | grep -A3 "type: " > > type: preonly > > maximum iterations=1, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=1. > > left preconditioning > > -- > > type: lu > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: external > > -- > > type: mumps > > rows=1050625, cols=1050625 > > package used to perform factorization: mumps > > total: nonzeros=115377847, allocated nonzeros=115377847 > > -- > > type: mpiaij > > rows=1050625, cols=1050625 > > total: nonzeros=7346177, allocated nonzeros=7346177 > > total number of mallocs used during MatSetValues calls=0 > > ``` > > > > I would greatly appreciate any insights you may have on this matter. > Thank you in advance for your time and assistance. > > > > Best wishes, > > Zongze > > > >
[petsc-users] Random Error of mumps: out of memory: INFOG(1)=-9
Hi,
I am writing to seek your advice regarding a problem I encountered while
using multigrid to solve a certain issue.
I am currently using multigrid with the coarse problem solved by PCLU.
However, the PC failed randomly with the error below (the value of INFO(2)
may differ):
```shell
[ 0] Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9,
INFO(2)=36
```
Upon checking the documentation of MUMPS, I discovered that increasing the
value of ICNTL(14) may help resolve the issue. Specifically, I set the
option -mat_mumps_icntl_14 to a higher value (such as 40), and the error
seemed to disappear after I set the value of ICNTL(14) to 80. However, I am
still curious as to why MUMPS failed randomly in the first place.
Upon further inspection, I found that the number of nonzeros of the PETSc
matrix and the MUMPS matrix were different every time I ran the code. I am
now left with the following questions:
1. What could be causing the number of nonzeros of the MUMPS matrix to
change every time I run the code?
2. Why is the number of nonzeros of the MUMPS matrix significantly greater
than that of the PETSc matrix (as seen in the output of ksp_view, 115025949
vs 7346177)?
3. Is it possible that the varying number of nonzeros of the MUMPS matrix
is the cause of the random failure?
I have attached a test example written in Firedrake. The output of
`ksp_view` after running the code twice is included below for your
reference.
In the output, the number of nonzeros of the MUMPS matrix was 115025949 and
115377847, respectively, while that of the PETSc matrix was only 7346177.
```shell
(complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view
::ascii_info_detail | grep -A3 "type: "
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=1.
left preconditioning
--
type: lu
out-of-place factorization
tolerance for zero pivot 2.22045e-14
matrix ordering: external
--
type: mumps
rows=1050625, cols=1050625
package used to perform factorization: mumps
total: nonzeros=115025949, allocated nonzeros=115025949
--
type: mpiaij
rows=1050625, cols=1050625
total: nonzeros=7346177, allocated nonzeros=7346177
total number of mallocs used during MatSetValues calls=0
(complex-int32-mkl) $ mpiexec -n 32 python test_mumps.py -ksp_view
::ascii_info_detail | grep -A3 "type: "
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=1.
left preconditioning
--
type: lu
out-of-place factorization
tolerance for zero pivot 2.22045e-14
matrix ordering: external
--
type: mumps
rows=1050625, cols=1050625
package used to perform factorization: mumps
total: nonzeros=115377847, allocated nonzeros=115377847
--
type: mpiaij
rows=1050625, cols=1050625
total: nonzeros=7346177, allocated nonzeros=7346177
total number of mallocs used during MatSetValues calls=0
```
I would greatly appreciate any insights you may have on this matter. Thank
you in advance for your time and assistance.
Best wishes,
Zongze
from firedrake import *
from firedrake.petsc import PETSc
rank, size = COMM_WORLD.rank, COMM_WORLD.size
opts = PETSc.Options()
N = opts.getInt('N', 32*size)
test_mesh = RectangleMesh(nx=N, ny=N, Lx=1, Ly=1) # Build mesh on the domain
x, y = SpatialCoordinate(test_mesh)
f = sin(pi*x)*sin(pi*y)
g = Constant(0)
V = FunctionSpace(test_mesh, 'CG', degree=1)# define FE space
u, v = TrialFunction(V), TestFunction(V)# define trial and test function
a = inner(grad(u), grad(v))*dx
L = inner(f, v)*dx # or f*v*dx
bc = DirichletBC(V, g=g, sub_domain='on_boundary')
u_h = Function(V, name='u_h')
solve(a == L, u_h, bcs=bc, options_prefix='') # We will introduce other ways to code in the following.
Re: [petsc-users] Inquiry regarding DMAdaptLabel function
Yes, It seems that firedrake only works with DMPlex. Thanks.
Best wishes,
Zongze
On Mon, 27 Feb 2023 at 22:53, Matthew Knepley wrote:
> On Mon, Feb 27, 2023 at 9:45 AM Zongze Yang wrote:
>
>> Hi, Matt
>>
>> Thanks for your clarification. Can I change the type of DMPlex to
>> DMForest?
>>
>
> You can, however DMForest is for structured adaptive meshes using
> quadtrees, and I do not believe
> Firedrake works with that.
>
> Thanks,
>
> Matt
>
>
>> Best wishes,
>> Zongze
>>
>>
>> On Mon, 27 Feb 2023 at 20:18, Matthew Knepley wrote:
>>
>>> On Sat, Feb 18, 2023 at 2:25 AM Zongze Yang
>>> wrote:
>>>
>>>> Dear PETSc Group,
>>>>
>>>> I am writing to inquire about the function DMAdaptLabel in PETSc.
>>>> I am trying to use it coarse a mesh, but the resulting mesh is refined.
>>>>
>>>> In the following code, all of the `adpat` label values were set to 2
>>>> (DM_ADAPT_COARSEN).
>>>> There must be something wrong. Could you give some suggestions?
>>>>
>>>
>>> Sorry for the late reply. You are right, I need to put in error messages
>>> for this. Here is what is happening.
>>> PETSc tries to fallback if you do not have certain packages. In this
>>> case, you are not using DMForest,
>>> which responds to both coarsen and refine, so the
>>> mesh generator interprets all markers as refine (they
>>> cannot coarsen). I will add a check that fails on the coarsen marker.
>>>
>>> Coarsening is much more difficult in the presence of boundaries, which
>>> is why it is not implemented in
>>> most packages. For unstructured coarsening, I do not think there is any
>>> choice but MMG.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> ```python
>>>> from firedrake import *
>>>> from firedrake.petsc import PETSc
>>>>
>>>> def mark_all_cells(mesh):
>>>> plex = mesh.topology_dm
>>>> with PETSc.Log.Event("ADD_ADAPT_LABEL"):
>>>> plex.createLabel('adapt')
>>>> cs, ce = plex.getHeightStratum(0)
>>>> for i in range(cs, ce):
>>>> plex.setLabelValue('adapt', i, 2)
>>>>
>>>> return plex
>>>>
>>>> mesh = RectangleMesh(10, 10, 1, 1)
>>>>
>>>> x = SpatialCoordinate(mesh)
>>>> V = FunctionSpace(mesh, 'CG', 1)
>>>> f = Function(V).interpolate(10 + 10*sin(x[0]))
>>>> triplot(mesh)
>>>>
>>>> plex = mark_all_cells(mesh)
>>>> new_plex = plex.adaptLabel('adapt')
>>>> mesh = Mesh(new_plex)
>>>> triplot(mesh)
>>>> ```
>>>>
>>>> Thank you very much for your time.
>>>>
>>>> Best wishes,
>>>> Zongze
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
Re: [petsc-users] Inquiry regarding DMAdaptLabel function
Hi, Matt,
I tested coarsening a mesh by using ParMMg without firedrake, and found
some issues:
see the code and results here:
https://gitlab.com/petsc/petsc/-/issues/1331
Could you have a look and give some comments or suggestions?
Best wishes,
Zongze
On Mon, 27 Feb 2023 at 20:19, Matthew Knepley wrote:
> On Sat, Feb 18, 2023 at 6:41 AM Zongze Yang wrote:
>
>> Another question on mesh coarsening is about `DMCoarsen` which will fail
>> when running in parallel.
>>
>> I generate a mesh in Firedrake, and then create function space and
>> functions, after that, I get the dmplex and coarsen it.
>> When running in serials, I get the mesh coarsened correctly. But it
>> failed with errors in ParMMG when running parallel.
>>
>> However, If I did not create function space and functions on the original
>> mesh, everything works fine too.
>>
>> The code and the error logs are attached.
>>
>
> I believe the problem is that Firedrake and PETSc currently have
> incompatible coordinate spaces. We are working
> to fix this, and I expect it to work by this summer.
>
> Thanks,
>
> Matt
>
>
>> Thank you for your time and attention。
>>
>> Best wishes,
>> Zongze
>>
>>
>> On Sat, 18 Feb 2023 at 15:24, Zongze Yang wrote:
>>
>>> Dear PETSc Group,
>>>
>>> I am writing to inquire about the function DMAdaptLabel in PETSc.
>>> I am trying to use it coarse a mesh, but the resulting mesh is refined.
>>>
>>> In the following code, all of the `adpat` label values were set to 2
>>> (DM_ADAPT_COARSEN).
>>> There must be something wrong. Could you give some suggestions?
>>>
>>> ```python
>>> from firedrake import *
>>> from firedrake.petsc import PETSc
>>>
>>> def mark_all_cells(mesh):
>>> plex = mesh.topology_dm
>>> with PETSc.Log.Event("ADD_ADAPT_LABEL"):
>>> plex.createLabel('adapt')
>>> cs, ce = plex.getHeightStratum(0)
>>> for i in range(cs, ce):
>>> plex.setLabelValue('adapt', i, 2)
>>>
>>> return plex
>>>
>>> mesh = RectangleMesh(10, 10, 1, 1)
>>>
>>> x = SpatialCoordinate(mesh)
>>> V = FunctionSpace(mesh, 'CG', 1)
>>> f = Function(V).interpolate(10 + 10*sin(x[0]))
>>> triplot(mesh)
>>>
>>> plex = mark_all_cells(mesh)
>>> new_plex = plex.adaptLabel('adapt')
>>> mesh = Mesh(new_plex)
>>> triplot(mesh)
>>> ```
>>>
>>> Thank you very much for your time.
>>>
>>> Best wishes,
>>> Zongze
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
Re: [petsc-users] Inquiry regarding DMAdaptLabel function
Hi, Matt
Thanks for your clarification. Can I change the type of DMPlex to DMForest?
Best wishes,
Zongze
On Mon, 27 Feb 2023 at 20:18, Matthew Knepley wrote:
> On Sat, Feb 18, 2023 at 2:25 AM Zongze Yang wrote:
>
>> Dear PETSc Group,
>>
>> I am writing to inquire about the function DMAdaptLabel in PETSc.
>> I am trying to use it coarse a mesh, but the resulting mesh is refined.
>>
>> In the following code, all of the `adpat` label values were set to 2
>> (DM_ADAPT_COARSEN).
>> There must be something wrong. Could you give some suggestions?
>>
>
> Sorry for the late reply. You are right, I need to put in error messages
> for this. Here is what is happening.
> PETSc tries to fallback if you do not have certain packages. In this case,
> you are not using DMForest,
> which responds to both coarsen and refine, so the
> mesh generator interprets all markers as refine (they
> cannot coarsen). I will add a check that fails on the coarsen marker.
>
> Coarsening is much more difficult in the presence of boundaries, which is
> why it is not implemented in
> most packages. For unstructured coarsening, I do not think there is any
> choice but MMG.
>
> Thanks,
>
> Matt
>
> ```python
>> from firedrake import *
>> from firedrake.petsc import PETSc
>>
>> def mark_all_cells(mesh):
>> plex = mesh.topology_dm
>> with PETSc.Log.Event("ADD_ADAPT_LABEL"):
>> plex.createLabel('adapt')
>> cs, ce = plex.getHeightStratum(0)
>> for i in range(cs, ce):
>> plex.setLabelValue('adapt', i, 2)
>>
>> return plex
>>
>> mesh = RectangleMesh(10, 10, 1, 1)
>>
>> x = SpatialCoordinate(mesh)
>> V = FunctionSpace(mesh, 'CG', 1)
>> f = Function(V).interpolate(10 + 10*sin(x[0]))
>> triplot(mesh)
>>
>> plex = mark_all_cells(mesh)
>> new_plex = plex.adaptLabel('adapt')
>> mesh = Mesh(new_plex)
>> triplot(mesh)
>> ```
>>
>> Thank you very much for your time.
>>
>> Best wishes,
>> Zongze
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
Re: [petsc-users] Save images of ksp_monitor and ksp_view_eigenvaluses with user defined names
Hi, Barry Thanks for the tip. One more question: how can I save the log (draw_lg) figure by using `draw:image:joe.ppm`? Thanks. Zongze Barry Smith 于2023年2月9日周四 05:35写道： > >It cannot be done using the default X windows monitor and -draw_save > because there is no way to distinguish the files for each sub window images. > >However, there is an alternative. > >-ksp_view_eigenvalues draw:image:jeff.ppm -viewer_view > -ksp_monitor_solution draw:image:joe.ppm > > This alternative only supports .ppm files (so you may need to call a > converter on the result) and does put each image in a separate file in its > own named directory, for example, joe/joe_0.ppm but at least it allows you > to have different named files. Of course you can also just run your code > twice with two different options. > >Unfortunately there is a bug in the KSP eigenmonitor viewing that I had > to fix to get this to work so you'll need to checkout the > *barry/2023-02-08/fix-ksp-monitor-eigenvalues-draw *branch of PETSc to > use the option I suggest. > > Barry > > > > > On Feb 8, 2023, at 5:09 AM, Zongze Yang wrote: > > Hi, PETSc group, > > I was trying to save figures of the residual and eigenvalues with > different names but not default names. > > The default name is used when I use `-draw_save .png`. All images are > saved. > ``` > python test.py -N 16 -test1_ksp_type gmres -test1_pc_type jacobi > -test1_ksp_view_eigenvalues draw -test1_ksp_monitor draw::draw_lg > -draw_save .png > ``` > But when I use `-draw_save abc.png`, only the figure of eigenvalues is > saved. > ``` > python test.py -N 16 -test1_ksp_type gmres -test1_pc_type jacobi > -test1_ksp_view_eigenvalues draw -test1_ksp_monitor draw::draw_lg > -draw_save .png > ``` > > How can I add the command line options, to specify different names for > those images? > > Thanks, > Zongze > > >
[petsc-users] Save images of ksp_monitor and ksp_view_eigenvaluses with user defined names
Hi, PETSc group, I was trying to save figures of the residual and eigenvalues with different names but not default names. The default name is used when I use `-draw_save .png`. All images are saved. ``` python test.py -N 16 -test1_ksp_type gmres -test1_pc_type jacobi -test1_ksp_view_eigenvalues draw -test1_ksp_monitor draw::draw_lg -draw_save .png ``` But when I use `-draw_save abc.png`, only the figure of eigenvalues is saved. ``` python test.py -N 16 -test1_ksp_type gmres -test1_pc_type jacobi -test1_ksp_view_eigenvalues draw -test1_ksp_monitor draw::draw_lg -draw_save .png ``` How can I add the command line options, to specify different names for those images? Thanks, Zongze
Re: [petsc-users] Build error with slepc: Unable to locate PETSc BAMG dynamic library
Thanks for all the suggestions. Will try to build with ` --download-slepc-configure-arguments='--with-slepc4py=1 --have-petsc4py=1'`. Thanks, Zongze Pierre Jolivet 于2022年11月17日周四 23:23写道： > > > > On 17 Nov 2022, at 4:11 PM, Satish Balay via petsc-users < > petsc-users@mcs.anl.gov> wrote: > > > >> --download-bamg --download-slepc > --download-slepc-configure-arguments=--with-slepc4py=1 > > > > I guess this won't really work > > It does work. > Just tried ./configure --download-slepc --download-bamg --with-petsc4py > '--download-slepc-configure-arguments=--with-slepc4py=1 --have-petsc4py=1' > --with-fc=0 > No issue whatsoever. > Matt, you should probably force that flag (--have-petsc4py=1) in bamg.py > (and you should change '+carg+'./configure to '+carg+self.python.pyexe+' > ./configure as in slepc.py) > > Thanks, > Pierre > > > as the order of build should be: > > - petsc > > - slepc > > - bamg > > - slepc4py > > > > And its not easy to do this via configure -without hacks. Currently the > above build has the order (hence fails): > > - petsc > > - slepc > > - slepc4py > > - bamg > > > > I guess the alternative is: build slepc4py separately after > petsc/slepc/bamg are built. > > > > Satish > > > > On Thu, 17 Nov 2022, Zongze Yang wrote: > > > >> Hello, I tried to build petsc with slepc. `make` give the following > error > >> information. How can I figure out the problem? The configure.log and > >> make.log are attached. > >> > >> ``` > >> *** Building SLEPc *** > >> Checking environment... done > >> Checking PETSc installation... done > >> Processing slepc4py... [0]PETSC ERROR: - Error > Message > >> -- > >> [0]PETSC ERROR: Unable to open file > >> [0]PETSC ERROR: Unable to locate PETSc BAMG dynamic library > >> You cannot move the dynamic libraries! > >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble > shooting. > >> [0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-291-g89fba64eb45 > >> GIT Date: 2022-11-15 14:31:39 + > >> [0]PETSC ERROR: Unknown Name on a arch-main-debug named ws6 by z2yang > Thu > >> Nov 17 21:59:09 2022 > >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-main-debug > >> --download-bamg --download-bison --download-chaco --download-ctetgen > >> --download-egads --download-eigen --download-exodusii --download-fftw > >> --download-hpddm --download-ks --download-libceed --download-metis > >> --download-ml --download-mmg --download-mumps --download-netcdf > >> --download-opencascade --download-p4est --download-parmetis > >> --download-parmmg --download-pnetcdf --download-pragmatic > >> --download-ptscotch --download-scalapack --download-slepc > >> --download-slepc-configure-arguments=--with-slepc4py=1 > >> --download-suitesparse --download-superlu_dist --download-tetgen > >> --download-triangle --download-cmake --download-hdf5 --download-mpich > >> --download-mpi4py --download-slepc --download-zlib --download-libpng > >> --download-muparser --with-petsc4py=1 --with-shared-libraries --with-x=1 > >> > --with-x-include="[/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/libx11-1.7.0-5c4ah77x6u7zfm6msg6hbkt23vmwjgkz/include,/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/xproto-7.0.31-z33ate5bew7b7xrpj3pv6nb3towcfimo/include,/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/kbproto-1.0.7-ea2l5e2kp43i2b423eotqxseywjvqis6/include,/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/libxcb-1.14-e2ea2x3zga5xipq5wvcgsw25ilq5yo63/include,/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/libxau-1.0.8-gmwxeffxcbkmxvwawtndhutiwficmxwv/include,/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/libxdmcp-1.1.2-bsggzn5pf6pu5guwbooi3riu5uhaqgee/include]" > >> > --with-x-lib="-L/home/z2yang/opt/spack/opt/spack/linux-ubuntu22.04-cascadelake/gcc-11.2.0/libx11-1.7.0-5c4ah77x6u7zfm6msg6hbkt23vmwjgkz/lib > >> -lX11" --force > >> [0]PETSC ERROR: #1 PetscInitialize_DynamicLibraries() at > >> /home/z2yang/repos/petsc/src/sys/dll/reg.c:135 > >> [0]PETSC ERROR: #2 PetscInitialize_Common() at > >> /home/z2yang/repos/petsc/src/sys/objects/pinit.c:1025 > >> [0]PETSC ERROR: #3 PetscInitialize() at > >> /home/z2yang/repos/petsc/src/sys/objects/pinit.c:1267 > >> Traceba
Re: [petsc-users] How to show the x window for cmd `make -f ./gmakefile test ...`?
Thanks! Matthew Knepley 于2022年10月6日周四 16:19写道： > On Thu, Oct 6, 2022 at 9:16 AM Zongze Yang wrote: > >> Hi, everyone, >> >> I am trying to run some test cases with x window, but the x window never >> showed up with command `make -f ./gmakefile test ...`. It seems a default >> option `-nox` is set. How to disable this option for `make test`? >> > > Yes, we disable it for tests by default in order to make the CI efficient. > You can edit > > $PETSC_DIR/$PETSC_ARCH/lib/petsc/conf/petscvariables > > and remove it from PETSC_TEST_OPTIONS, which should be the last line. > > Thanks, > > Matt > > >> An example is shown below: >> ``` >> z2yang@ws6:~/repos/petsc$ PETSC_ARCH=arch-main-debug make -f ./gmakefile >> test search="dm_impls_plex_tests-ex20_2d" TIMEOUT=5000 >> EXTRA_OPTIONS="-post_adapt_dm_view draw:x -draw_pause -1 -options_view >> -petsc_ci false" >> Using MAKEFLAGS: -- EXTRA_OPTIONS=-post_adapt_dm_view draw:x -draw_pause >> -1 -options_view -petsc_ci false TIMEOUT=5000 >> search=dm_impls_plex_tests-ex20_2d >> TEST >> arch-main-debug/tests/counts/dm_impls_plex_tests-ex20_2d.counts >> ok dm_impls_plex_tests-ex20_2d >> not ok diff-dm_impls_plex_tests-ex20_2d # Error code: 1 >> # 12,22c12,26 >> # < DM Object: Post Adaptation Mesh 1 MPI process >> # < type: plex >> # < Post Adaptation Mesh in 2 dimensions: >> # < Number of 0-cells per rank: 49 >> # < Number of 1-cells per rank: 120 >> # < Number of 2-cells per rank: 72 >> # < Labels: >> # < celltype: 3 strata with value/size (1 (120), 3 (72), 0 (49)) >> # < depth: 3 strata with value/size (0 (49), 1 (120), 2 (72)) >> # < marker: 1 strata with value/size (1 (48)) >> # < Face Sets: 1 strata with value/size (1 (36)) >> # --- >> # > #PETSc Option Table entries: >> # > -check_pointer_intensity 0 >> # > -dm_coord_space 0 >> # > -dm_plex_box_faces 3,3 >> # > -draw_pause -1 >> # > -error_output_stdout >> # > -malloc_dump >> # > -nox >> # > -nox_warning >> # > -options_view >> # > -petsc_ci false >> # > -post_adapt_dm_view draw:x >> # > -pre_adapt_dm_view ascii::ascii_info >> # > -use_gpu_aware_mpi 0 >> # > #End of PETSc Option Table entries >> >> >> # FAILED diff-dm_impls_plex_tests-ex20_2d >> # >> # To rerun failed tests: >> # /usr/bin/gmake -f gmakefile test test-fail=1 >> ``` >> >> Thanks, >> Zongze >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
[petsc-users] How to show the x window for cmd `make -f ./gmakefile test ...`?
Hi, everyone, I am trying to run some test cases with x window, but the x window never showed up with command `make -f ./gmakefile test ...`. It seems a default option `-nox` is set. How to disable this option for `make test`? An example is shown below: ``` z2yang@ws6:~/repos/petsc$ PETSC_ARCH=arch-main-debug make -f ./gmakefile test search="dm_impls_plex_tests-ex20_2d" TIMEOUT=5000 EXTRA_OPTIONS="-post_adapt_dm_view draw:x -draw_pause -1 -options_view -petsc_ci false" Using MAKEFLAGS: -- EXTRA_OPTIONS=-post_adapt_dm_view draw:x -draw_pause -1 -options_view -petsc_ci false TIMEOUT=5000 search=dm_impls_plex_tests-ex20_2d TEST arch-main-debug/tests/counts/dm_impls_plex_tests-ex20_2d.counts ok dm_impls_plex_tests-ex20_2d not ok diff-dm_impls_plex_tests-ex20_2d # Error code: 1 # 12,22c12,26 # < DM Object: Post Adaptation Mesh 1 MPI process # < type: plex # < Post Adaptation Mesh in 2 dimensions: # < Number of 0-cells per rank: 49 # < Number of 1-cells per rank: 120 # < Number of 2-cells per rank: 72 # < Labels: # < celltype: 3 strata with value/size (1 (120), 3 (72), 0 (49)) # < depth: 3 strata with value/size (0 (49), 1 (120), 2 (72)) # < marker: 1 strata with value/size (1 (48)) # < Face Sets: 1 strata with value/size (1 (36)) # --- # > #PETSc Option Table entries: # > -check_pointer_intensity 0 # > -dm_coord_space 0 # > -dm_plex_box_faces 3,3 # > -draw_pause -1 # > -error_output_stdout # > -malloc_dump # > -nox # > -nox_warning # > -options_view # > -petsc_ci false # > -post_adapt_dm_view draw:x # > -pre_adapt_dm_view ascii::ascii_info # > -use_gpu_aware_mpi 0 # > #End of PETSc Option Table entries # FAILED diff-dm_impls_plex_tests-ex20_2d # # To rerun failed tests: # /usr/bin/gmake -f gmakefile test test-fail=1 ``` Thanks, Zongze
Re: [petsc-users] Is the results of `DMAdaptLabel` as expected in `src/dm/impls/plex/tests/ex20.c`
Matthew Knepley 于2022年10月5日周三 00:33写道： > On Tue, Oct 4, 2022 at 3:19 PM Zongze Yang wrote: > >> Hi everyone, >> >> I am learning how to use the `DMAdaptLabel` for `DMPlex`, and found the >> example `src/dm/impls/plex/tests/ex20.c` which label one cell to refine. >> >> 1. This example is just a uniform refinement when using the following >> command. (see attached pdfs for the results). >> ``` >> [real-int32-gcc] >> z2yang@ws5:~/opt/firedrake/real-int32-gcc/petsc/src/dm/impls/plex/tests$ >> ./ex20 -dm_plex_box_faces 3,3 -dm_coord_space 0 -pre_adapt_dm_view >> ascii::ascii_info -post_adapt_dm_view draw:tikz:figure2.tex >> ``` >> Is this expected for this example? >> > > Hi Zongze, > > Yes, I agree this is not easy to see. If you give -dm_plex_transform_view, > you can see the kind of transform being used > > knepley/pylith $:/PETSc3/petsc/petsc-pylith$ PETSC_ARCH=arch-pylith-opt > make -j8 -f ./gmakefile test search="dm_impls_plex_tests-ex20_ > 2d" TIMEOUT=5000 EXTRA_OPTIONS="-dm_plex_transform_view" > Using MAKEFLAGS: --jobserver-fds=3,4 -j -- > EXTRA_OPTIONS=-dm_plex_transform_view TIMEOUT=5000 > search=dm_impls_plex_tests-ex20_2d > TEST > arch-pylith-opt/tests/counts/dm_impls_plex_tests-ex20_2d.counts > ok dm_impls_plex_tests-ex20_2d > not ok diff-dm_impls_plex_tests-ex20_2d # Error code: 1 > # 11a12,14 > # > DMPlexTransform Object: 1 MPI process > # > type: refine_regular > # > Regular refinement DMPlexTransform_0x8400_1 > > You can see that it is regular refinement, so it ignores the input and > refines everything. If you change it, you can get adaptive refinement, > > knepley/pylith $:/PETSc3/petsc/petsc-pylith$ PETSC_ARCH=arch-pylith-opt > make -j8 -f ./gmakefile test search="dm_impls_plex_tests-ex20_ > 2d" TIMEOUT=5000 EXTRA_OPTIONS="-pre_adapt_dm_view draw > -post_adapt_dm_view draw -draw_pause -1 -dm_plex_transform_type refine_sbr" > > I attached the plot. > > Hi Matt, Thanks for your clear explanation. Now, I see that by setting different transform types I can refine the mesh by different algorithms. But why the refinement algorithms are classified as transform? 2. I found there is a function named `DMAdaptLabel_Plex`, and >> `DMAdaptLabel` did not call that function when the type of the dm is >> `DMPlex`. Is the function `DMAdaptLabel_Plex` still in use? >> > > No. I rewrote all the transformations last year. I think the new form is > much smaller, cleaner, and more performant. I should delete this function, > but I am finishing > up the review of all adaptive refinement with Joe Wallwork at Imperial. > > >> 3. `DMAdaptLabel` seems to lack some useful information when I use the >> wrong adaptor. For example, if I set `-dm_adaptor mmg`, then the process >> will give a segment fault because the `metric` passed to >> `DMAdaptMetric_Mmg_Plex` is NULL, see the output below: >> ``` >> [real-int32-gcc] >> z2yang@ws5:~/opt/firedrake/real-int32-gcc/petsc/src/dm/impls/plex/tests$ >> ./ex20 -dm_plex_box_faces 3,3 -dm_coord_space 0 -pre_adapt_dm_view >> draw:tikz:figure1.tex -post_adapt_dm_view draw:tikz:figure2.tex -dm_adaptor >> mmg >> [0]PETSC ERROR: >> >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and >> https://petsc.org/release/faq/ >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is >> causing the crash. >> Abort(59) on node 0 (rank 0 in comm 0): application called >> MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> ``` >> > > Hmm, I at least get an error message: > > # [0]PETSC ERROR: - Error Message > -- > # [0]PETSC ERROR: Null argument, when expecting valid pointer > # [0]PETSC ERROR: Null Pointer: Parameter # 1 > # [0]PETSC ERROR: WARNING! There are option(s) set that were not > used! Could be the program crashed before they were used or a spelling > mistake, etc! > # [0]PETSC ERROR: Option left: name:-post_adapt_dm_view value: > ascii::ascii_info > # [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble > shooting.
[petsc-users] Is the results of `DMAdaptLabel` as expected in `src/dm/impls/plex/tests/ex20.c`
Hi everyone, I am learning how to use the `DMAdaptLabel` for `DMPlex`, and found the example `src/dm/impls/plex/tests/ex20.c` which label one cell to refine. 1. This example is just a uniform refinement when using the following command. (see attached pdfs for the results). ``` [real-int32-gcc] z2yang@ws5:~/opt/firedrake/real-int32-gcc/petsc/src/dm/impls/plex/tests$ ./ex20 -dm_plex_box_faces 3,3 -dm_coord_space 0 -pre_adapt_dm_view ascii::ascii_info -post_adapt_dm_view draw:tikz:figure2.tex ``` Is this expected for this example? 2. I found there is a function named `DMAdaptLabel_Plex`, and `DMAdaptLabel` did not call that function when the type of the dm is `DMPlex`. Is the function `DMAdaptLabel_Plex` still in use? 3. `DMAdaptLabel` seems to lack some useful information when I use the wrong adaptor. For example, if I set `-dm_adaptor mmg`, then the process will give a segment fault because the `metric` passed to `DMAdaptMetric_Mmg_Plex` is NULL, see the output below: ``` [real-int32-gcc] z2yang@ws5:~/opt/firedrake/real-int32-gcc/petsc/src/dm/impls/plex/tests$ ./ex20 -dm_plex_box_faces 3,3 -dm_coord_space 0 -pre_adapt_dm_view draw:tikz:figure1.tex -post_adapt_dm_view draw:tikz:figure2.tex -dm_adaptor mmg [0]PETSC ERROR: [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ``` Thanks, Zongze Yang pre_adapt.pdf Description: Adobe PDF document post_adapt.pdf Description: Adobe PDF document
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
Thank you for your reply. May I ask for some references on the order of the dofs on PETSc's FE Space (especially high order elements)? Thanks, Zongze Matthew Knepley 于2022年6月18日周六 20:02写道： > On Sat, Jun 18, 2022 at 2:16 AM Zongze Yang wrote: > >> In order to check if I made mistakes in the python code, I try to use c >> code to show the issue on DMProjectCoordinates. The code and mesh file is >> attached. >> If the code is correct, there must be something wrong with >> `DMProjectCoordinates` or `DMPlexCreateGmshFromFile` for high-order mesh. >> > > Something is definitely wrong with high order, periodic simplices from > Gmsh. We had not tested that case. I am at a conference and cannot look at > it for a week. > My suspicion is that the space we make when reading in the Gmsh > coordinates does not match the values (wrong order). > > Thanks, > > Matt > > >> The command and the output are listed below: (Obviously the bounding box >> is changed.) >> ``` >> $ ./test_gmsh_load_2rd -filename cube-p2.msh -old_fe_view -new_fe_view >> Old Bounding Box: >> 0: lo = 0. hi = 1. >> 1: lo = 0. hi = 1. >> 2: lo = 0. hi = 1. >> PetscFE Object: OldCoordinatesFE 1 MPI processes >> type: basic >> Basic Finite Element in 3 dimensions with 3 components >> PetscSpace Object: P2 1 MPI processes >> type: sum >> Space in 3 variables with 3 components, size 30 >> Sum space of 3 concatenated subspaces (all identical) >> PetscSpace Object: sum component (sumcomp_) 1 MPI processes >> type: poly >> Space in 3 variables with 1 components, size 10 >> Polynomial space of degree 2 >> PetscDualSpace Object: P2 1 MPI processes >> type: lagrange >> Dual space with 3 components, size 30 >> Discontinuous Lagrange dual space >> Quadrature of order 5 on 27 points (dim 3) >> PetscFE Object: NewCoordinatesFE 1 MPI processes >> type: basic >> Basic Finite Element in 3 dimensions with 3 components >> PetscSpace Object: P2 1 MPI processes >> type: sum >> Space in 3 variables with 3 components, size 30 >> Sum space of 3 concatenated subspaces (all identical) >> PetscSpace Object: sum component (sumcomp_) 1 MPI processes >> type: poly >> Space in 3 variables with 1 components, size 10 >> Polynomial space of degree 2 >> PetscDualSpace Object: P2 1 MPI processes >> type: lagrange >> Dual space with 3 components, size 30 >> Continuous Lagrange dual space >> Quadrature of order 5 on 27 points (dim 3) >> New Bounding Box: >> 0: lo = 2.5624e-17 hi = 8. >> 1: lo = -9.23372e-17 hi = 7. >> 2: lo = 2.72091e-17 hi = 8.5 >> ``` >> >> Thanks, >> Zongze >> >> Zongze Yang 于2022年6月17日周五 14:54写道： >> >>> I tried the projection operation. However, it seems that the projection >>> gives the wrong solution. After projection, the bounding box is changed! >>> See logs below. >>> >>> First, I patch the petsc4py by adding `DMProjectCoordinates`: >>> ``` >>> diff --git a/src/binding/petsc4py/src/PETSc/DM.pyx >>> b/src/binding/petsc4py/src/PETSc/DM.pyx >>> index d8a58d183a..dbcdb280f1 100644 >>> --- a/src/binding/petsc4py/src/PETSc/DM.pyx >>> +++ b/src/binding/petsc4py/src/PETSc/DM.pyx >>> @@ -307,6 +307,12 @@ cdef class DM(Object): >>> PetscINCREF(c.obj) >>> return c >>> >>> +def projectCoordinates(self, FE fe=None): >>> +if fe is None: >>> +CHKERR( DMProjectCoordinates(self.dm, NULL) ) >>> +else: >>> +CHKERR( DMProjectCoordinates(self.dm, fe.fe) ) >>> + >>> def getBoundingBox(self): >>> cdef PetscInt i,dim=0 >>> CHKERR( DMGetCoordinateDim(self.dm, ) ) >>> diff --git a/src/binding/petsc4py/src/PETSc/petscdm.pxi >>> b/src/binding/petsc4py/src/PETSc/petscdm.pxi >>> index 514b6fa472..c778e39884 100644 >>> --- a/src/binding/petsc4py/src/PETSc/petscdm.pxi >>> +++ b/src/binding/petsc4py/src/PETSc/petscdm.pxi >>> @@ -90,6 +90,7 @@ cdef extern from * nogil: >>> int DMGetCoordinateDim(PetscDM,PetscInt*) >>> int DMSetCoordinateDim(PetscDM,PetscInt) >>> int DMLocalizeCoordinates(PetscDM) >>> +int DMProjectCoordinates(PetscDM, PetscFE) >>> >>> int DMCreateInterpolation(PetscDM,PetscDM,PetscMat*,PetscVec*) >>> int D
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
I tried the projection operation. However, it seems that the projection
gives the wrong solution. After projection, the bounding box is changed!
See logs below.
First, I patch the petsc4py by adding `DMProjectCoordinates`:
```
diff --git a/src/binding/petsc4py/src/PETSc/DM.pyx
b/src/binding/petsc4py/src/PETSc/DM.pyx
index d8a58d183a..dbcdb280f1 100644
--- a/src/binding/petsc4py/src/PETSc/DM.pyx
+++ b/src/binding/petsc4py/src/PETSc/DM.pyx
@@ -307,6 +307,12 @@ cdef class DM(Object):
PetscINCREF(c.obj)
return c
+def projectCoordinates(self, FE fe=None):
+if fe is None:
+CHKERR( DMProjectCoordinates(self.dm, NULL) )
+else:
+CHKERR( DMProjectCoordinates(self.dm, fe.fe) )
+
def getBoundingBox(self):
cdef PetscInt i,dim=0
CHKERR( DMGetCoordinateDim(self.dm, ) )
diff --git a/src/binding/petsc4py/src/PETSc/petscdm.pxi
b/src/binding/petsc4py/src/PETSc/petscdm.pxi
index 514b6fa472..c778e39884 100644
--- a/src/binding/petsc4py/src/PETSc/petscdm.pxi
+++ b/src/binding/petsc4py/src/PETSc/petscdm.pxi
@@ -90,6 +90,7 @@ cdef extern from * nogil:
int DMGetCoordinateDim(PetscDM,PetscInt*)
int DMSetCoordinateDim(PetscDM,PetscInt)
int DMLocalizeCoordinates(PetscDM)
+int DMProjectCoordinates(PetscDM, PetscFE)
int DMCreateInterpolation(PetscDM,PetscDM,PetscMat*,PetscVec*)
int DMCreateInjection(PetscDM,PetscDM,PetscMat*)
```
Then in python, I load a mesh and project the coordinates to P2:
```
import firedrake as fd
from firedrake.petsc import PETSc
# plex = fd.mesh._from_gmsh('test-fd-load-p2.msh')
plex = fd.mesh._from_gmsh('test-fd-load-p2-rect.msh')
print('old bbox:', plex.getBoundingBox())
dim = plex.getDimension()
# (dim, nc, isSimplex, k, qorder,
comm=None)
fe_new = PETSc.FE().createLagrange(dim, dim, True, 2, PETSc.DETERMINE)
plex.projectCoordinates(fe_new)
fe_new.view()
print('new bbox:', plex.getBoundingBox())
```
The output is (The bounding box is changed!)
```
old bbox: ((0.0, 1.0), (0.0, 1.0), (0.0, 1.0))
PetscFE Object: P2 1 MPI processes
type: basic
Basic Finite Element in 3 dimensions with 3 components
PetscSpace Object: P2 1 MPI processes
type: sum
Space in 3 variables with 3 components, size 30
Sum space of 3 concatenated subspaces (all identical)
PetscSpace Object: sum component (sumcomp_) 1 MPI processes
type: poly
Space in 3 variables with 1 components, size 10
Polynomial space of degree 2
PetscDualSpace Object: P2 1 MPI processes
type: lagrange
Dual space with 3 components, size 30
Continuous Lagrange dual space
Quadrature of order 5 on 27 points (dim 3)
new bbox: ((-6.530133708576188e-17, 36.30670832662781),
(-3.899962995254311e-17, 36.2406171632539), (-8.8036464152166e-17,
36.111577025012224))
```
By the way, for the original DG coordinates, where can I find the
relation of the closure and the order of the dofs for the cell?
Thanks!
Zongze
Matthew Knepley 于2022年6月17日周五 01:11写道：
> On Thu, Jun 16, 2022 at 12:06 PM Zongze Yang wrote:
>
>>
>>
>> 在 2022年6月16日，23:22，Matthew Knepley 写道：
>>
>>
>> On Thu, Jun 16, 2022 at 11:11 AM Zongze Yang
>> wrote:
>>
>>> Hi, if I load a `gmsh` file with second-order elements, the coordinates
>>> will be stored in a DG-P2 space. After obtaining the coordinates of a cell,
>>> how can I map the coordinates to vertex and edge?
>>>
>>
>> By default, they are stored as P2, not DG.
>>
>>
>> I checked the coordinates vector, and found the dogs only defined on cell
>> other than vertex and edge, so I said they are stored as DG.
>> Then the function DMPlexVecGetClosure
>> <https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexVecGetClosure/> seems
>> return
>> the coordinates in lex order.
>>
>> Some code in reading gmsh file reads that
>>
>>
>> 1756: if (isSimplex) continuity = PETSC_FALSE
>> <https://petsc.org/main/docs/manualpages/Sys/PETSC_FALSE/>; /* XXX FIXME
>> Requires DMPlexSetClosurePermutationLexicographic() */
>>
>>
>> 1758: GmshCreateFE(comm, NULL, isSimplex, continuity, nodeType, dim,
>> coordDim, order, )
>>
>>
>> The continuity is set to false for simplex.
>>
>
> Oh, yes. That needs to be fixed. For now, you can just project it to P2 if
> you want using
>
> https://petsc.org/main/docs/manualpages/DM/DMProjectCoordinates/
>
> Thanks,
>
> Matt
>
>
>> Thanks,
>> Zongze
>>
>> You can ask for the coordinates of a vertex or an edge directly using
>>
>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexPointLocalRead/
>>
>> by giving the vertex or edge point. You can get all the coo
Re: [petsc-users] How to find the map between the high order coordinates of DMPlex and vertex numbering?
> 在 2022年6月16日，23:22，Matthew Knepley 写道：
>
>
>> On Thu, Jun 16, 2022 at 11:11 AM Zongze Yang wrote:
>
>> Hi, if I load a `gmsh` file with second-order elements, the coordinates will
>> be stored in a DG-P2 space. After obtaining the coordinates of a cell, how
>> can I map the coordinates to vertex and edge?
>
> By default, they are stored as P2, not DG.
I checked the coordinates vector, and found the dogs only defined on cell other
than vertex and edge, so I said they are stored as DG.
Then the function DMPlexVecGetClosure seems return the coordinates in lex order.
Some code in reading gmsh file reads that
1756: if (isSimplex) continuity = PETSC_FALSE; /* XXX FIXME Requires
DMPlexSetClosurePermutationLexicographic() */
1758: GmshCreateFE(comm, NULL, isSimplex, continuity, nodeType, dim,
coordDim, order, )
The continuity is set to false for simplex.
Thanks,
Zongze
>
> You can ask for the coordinates of a vertex or an edge directly using
>
> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexPointLocalRead/
>
> by giving the vertex or edge point. You can get all the coordinates on a
> cell, in the closure order, using
>
> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexVecGetClosure/
> Thanks,
>
> Matt
>
>> Below is some code load the gmsh file, I want to know the relation between
>> `cl` and `cell_coords`.
>>
>> ```
>> import firedrake as fd
>> import numpy as np
>>
>> # Load gmsh file (2rd)
>> plex = fd.mesh._from_gmsh('test-fd-load-p2-rect.msh')
>>
>> cs, ce = plex.getHeightStratum(0)
>>
>> cdm = plex.getCoordinateDM()
>> csec = dm.getCoordinateSection()
>> coords_gvec = dm.getCoordinates()
>>
>> for i in range(cs, ce):
>> cell_coords = cdm.getVecClosure(csec, coords_gvec, i)
>> print(f'coordinates for cell {i} :\n{cell_coords.reshape([-1, 3])}')
>> cl = dm.getTransitiveClosure(i)
>> print('closure:', cl)
>> break
>> ```
>>
>> Best wishes,
>> Zongze
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
