Hi Satish and Barry, Thank you very for getting back to me with the suggestion of the OMP_NUM_THREADS environmental variable and how to set the option no_signal_handler per Barry's suggestion.
Yes, I am using threaded Intel MKL. So, I first set the environmental variable OMP_NUM_THREADS=1, and run the Visual Studio debugger. It still crashes at the same place with the same errors. I then set the PETSC_OPTIONS=-no_signal_handler, the debugger refuses to pass through the access exception, and I have to stop the debugger; consequently no PETSc errors were reported. I know there is an option somewhere in the VS debugger to bypass this access exception, but I didn't pursue further. Finally, I removed Intel MKL 2000 (to uninstall it requires a 3rd party software!), and use Intel oneAPI MKL (2023) with PETSc 3.18.6 and Visual Studio 2022. Now my code with 1, 2 or 4 cores work! I will try 3.19.4 next and will keep you posted. Thank you again for your help, Thuc Bui ////////////////////////////////////////////////////////////////////////////////////////////////////////////////// ./configure --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-openmp \ --with-mpi-include='/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/MSMPI/latest/include/x64' \ --with-mpi-lib=[/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib] \ --with-blas-lapack-lib=[/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/IntelMKL/latest/lib64/mkl_intel_lp64.lib,/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/IntelMKL/latest/lib64/mkl_core.lib,/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/IntelMKL/latest/lib64/mkl_intel_thread.lib,/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/IntelMKL/latest/lib64/mkl_rt.lib,/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/IntelMKL/latest/lib64/libiomp5md.lib] \ --with-mpiexec='/cygdrive/d/Users/bbwannabe/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec' \ --with-shared-libraries=0 /////////////////////////////////////////////////////////////////////////////////////////////////////////////////// -----Original Message----- From: Satish Balay [mailto:ba...@mcs.anl.gov] Sent: Tuesday, September 19, 2023 10:01 AM To: Thuc Bui Cc: 'Barry Smith'; 'PETSc users list' Subject: Re: [petsc-users] Problem with BLASdot in 3.19.4 Its a run time option to petsc (application) binary. So you can either specify it via command line - at run time - or add it to env variable "PETSC_OPTIONS" - or add it to $HOME/.petscrc file Satish On Tue, 19 Sep 2023, Thuc Bui wrote: > Hi Barry, > > > > Thanks for getting back to me. The diagnostics were generated when tracing > under the VS debugger. To use the option –no_signal_handler, I believe I will > need to reconfigure PETSc with this additional option. I will try it now. > > > > Thuc > > > > > > From: Barry Smith [mailto:bsm...@petsc.dev] > Sent: Tuesday, September 19, 2023 8:24 AM > To: Thuc Bui > Cc: PETSc users list > Subject: Re: [petsc-users] Problem with BLASdot in 3.19.4 > > > > > > Can you run in the Microsoft Visual Studio debugger? Use the additional > PETSc option -no_signal_handler > > > > > > It won't show exactly where the SEGV happens but might focus in a bit on > it. For example it may be ddot() or ddot_(). > > > > Barry > > > > > > > > > > On Sep 19, 2023, at 2:04 AM, Thuc Bui <b...@calcreek.com> wrote: > > > > Hi Barry, > > > > Visual Studio 2022 is the problem! The code linked to Petsc 3.18.6 built with > VS 2022 also crashes at the same place. The same errors are shown below. I > don’t remember for sure, but I don’t think I was able to configure Petsc > 3.19.4 with VS 2019. However, I will still try that next. > > > > Thanks for your help, > > Thuc > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see <https://petsc.org/release/faq/#valgrind> > https://petsc.org/release/faq/#valgrind and <https://petsc.org/release/faq/> > https://petsc.org/release/faq/ > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > > [0]PETSC ERROR: #1 BLASdot() > > [0]PETSC ERROR: #2 VecNorm_Seq() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\impls\seq\bvec2.c:216 > > [0]PETSC ERROR: #3 VecNorm() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\interface\rvector.c:237 > > [0]PETSC ERROR: #4 VecNormalize() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\interface\rvector.c:318 > > [0]PETSC ERROR: #5 KSPGMRESCycle() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\impls\gmres\gmres.c:111 > > [0]PETSC ERROR: #6 KSPSolve_GMRES() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\impls\gmres\gmres.c:228 > > [0]PETSC ERROR: #7 KSPSolve_Private() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\interface\itfunc.c:899 > > [0]PETSC ERROR: #8 KSPSolve() at > D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\interface\itfunc.c:1071 > > > > job aborted: > > [ranks] message > > > > [0] application aborted > > aborting MPI_COMM_WORLD (comm=0x44000000), error 59, comm rank 0 > > > > > > > > From: petsc-users [ <mailto:petsc-users-boun...@mcs.anl.gov> > mailto:petsc-users-boun...@mcs.anl.gov] On Behalf Of Thuc Bui > Sent: Monday, September 18, 2023 4:24 PM > To: 'Barry Smith' > Cc: 'PETSc users list' > Subject: Re: [petsc-users] Problem with BLASdot in 3.19.4 > > > > Thanks a lot Barry, for getting back to me. Will do what you have suggested, > and get back with the results. > > > > Best regards, > > Thuc > > > > From: Barry Smith [ <mailto:bsm...@petsc.dev> mailto:bsm...@petsc.dev] > Sent: Monday, September 18, 2023 3:43 PM > To: Thuc Bui > Cc: PETSc users list > Subject: Re: [petsc-users] Problem with BLASdot in 3.19.4 > > > > > > Ok, two things are being changed at the same time: the version of PETSc > and the version of Visual Studio. > > > > Could you please try with the new Visual Studio version but the same older > PETSc version? If that works could you try with the old Visual Studio version > but the new PETSc version? > > > > Barry > > > > > > > > On Sep 18, 2023, at 6:26 PM, Thuc Bui < <mailto:b...@calcreek.com> > b...@calcreek.com> wrote: > > > > Dear Petsc users and experts, > > > > If someone can direct me how to track this bug, I would really appreciate it. > > > > The Petsc DLL library version 3.19.4 was built on Windows 10 with Visual > Studio 2022, and with Microsoft MPI 10.1.2 and Intel MKL 2020.3.279. The same > code works fine with Petsc 3.18.6 using the same versions of MS MPI and Intel > MKL, and built with Visual Studio 2019. > > > > When my code calls PetscCall(KSPSolve(...)), it gets to > PetscCall(KSPGMRESCycle(...)), PetscCall(VecNormalize(VEC_VV(0), &res)), > PetscCall(VecNorm(x, NORM_2, &norm)) and then crashes in VecNorm_Seq() at: > > > > PetscCallBLAS("BLASdot", ztmp[type == NORM_1_AND_2] = > PetscSqrtReal(PetscRealPart(BLASdot_(&bn, xx, &one, xx, &one)))); > > > > I tried to step into BLASdot_, but was unable to. I assume BLASdot_ belong to > Intel MKL, and its library does not contain debug information. After the code > exits, the errors are shown below the hash line. > > > > Should I install the latest Intel MKL to work with Petsc 3.19.4? > > > > Many thanks in advance for your help, > > Thuc Bui > > Senior R&D Engineer > > Calabazas Creek Research, Inc > > (650) 948-5361 > > > > > > ############################################### > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see <https://petsc.org/release/faq/#valgrind> > https://petsc.org/release/faq/#valgrind and <https://petsc.org/release/faq/> > https://petsc.org/release/faq/ > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > > [0]PETSC ERROR: #1 BLASdot() > > [0]PETSC ERROR: #2 VecNorm_Seq() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\vec\vec\impls\seq\bvec2.c:213 > > [0]PETSC ERROR: #3 VecNorm() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\vec\vec\interface\rvector.c:237 > > [0]PETSC ERROR: #4 VecNormalize() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\vec\vec\interface\rvector.c:318 > > [0]PETSC ERROR: #5 KSPGMRESCycle() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:110 > > [0]PETSC ERROR: #6 KSPSolve_GMRES() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 > > [0]PETSC ERROR: #7 KSPSolve_Private() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\ksp\ksp\interface\itfunc.c:898 > > [0]PETSC ERROR: #8 KSPSolve() at > D:\Users\bbwannabe\Documents\Petsc\latest\src\ksp\ksp\interface\itfunc.c:1070 > > > > job aborted: > > [ranks] message > > > > [0] application aborted > > aborting MPI_COMM_WORLD (comm=0x44000000), error 59, comm rank 0 > > > >
