On Tue, Apr 29, 2025 at 6:50 AM Klaij, Christiaan <c.kl...@marin.nl> wrote:
> Here's a slightly better error message, obtained --with-debugging=1
>
Is it possible that you have a mismatched EventBegin()/EventEnd() in your
code? That could be why we cannot reproduce it here.
Thanks,
Matt
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code
> lines) on different processors
> [0]PETSC ERROR: See
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4LwmzLRI$
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025
> [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on
> marclus3login2 by cklaij Tue Apr 29 12:43:54 2025
> [0]PETSC ERROR: Configure options:
> --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs
> --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0
> --with-debugging=1 --download-superlu_dist=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4MX1qQCC$
>
> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0
> --download-parmetis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4NVa8uiZ$
>
> --download-metis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4CbREwQV$
>
> --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild
> --with-ssl=0 --with-shared-libraries=1
> [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289
> [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379
> [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420
> [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443
> [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405
> [0]PETSC ERROR: #8 PetscLogHandlerView() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342
> [0]PETSC ERROR: #9 PetscLogView() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040
> [0]PETSC ERROR: #10
> /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130
>
> ________________________________________
> dr. ir. Christiaan Klaij
> | senior researcher | Research & Development | CFD Development
> T +31 317 49 33 44 <+31%20317%2049%2033%2044> | c.kl...@marin.nl |
> https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4JYE0ch-$
>
> [image: Facebook]
> <https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4G2Xsfs9$
> >
> [image: LinkedIn]
> <https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4GcxXOHk$
> >
> [image: YouTube]
> <https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4AzBikZl$
> >
>
> From: Klaij, Christiaan <c.kl...@marin.nl>
> Sent: Monday, April 28, 2025 3:53 PM
> To: Matthew Knepley
> Cc: Junchao Zhang; petsc-users@mcs.anl.gov; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> Bisecting would be quite hard, it's not just the petsc version that
> changed, also other libs, compilers, even os components.
>
> Chris
>
> ________________________________________
> From: Matthew Knepley <knep...@gmail.com>
> Sent: Monday, April 28, 2025 3:06 PM
> To: Klaij, Christiaan
> Cc: Junchao Zhang; petsc-users@mcs.anl.gov; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> You don't often get email from knep...@gmail.com. Learn why this is
> important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4Me9_aLp$
> >
> On Mon, Apr 28, 2025 at 8:45 AM Klaij, Christiaan via petsc-users <
> petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>> wrote:
> I've tried adding a nested log viewer to src/snes/tutorials/ex70.c,
> but it does not replicate the problem and works fine.
>
> Perhaps it is related to fortran, since the manualpage of
> PetscLogNestedBegin says "No fortran support" (why? we've been
> using it in fortran ever since).
>
> Therefore I've tried adding it to src/snes/ex5f90.F90 and that
> also works fine. It seems I cannot replicate the problem in a
> small example, unfortunately.
>
> We cannot replicate it here. Is there a chance you could bisect to see
> what change is responsible?
>
> Thanks,
>
> Matt
>
> Chris
>
> ________________________________________
> From: Junchao Zhang <junchao.zh...@gmail.com<mailto:
> junchao.zh...@gmail.com>>
> Sent: Saturday, April 26, 2025 3:51 PM
> To: Klaij, Christiaan
> Cc: petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> You don't often get email from junchao.zh...@gmail.com<mailto:
> junchao.zh...@gmail.com>. Learn why this is important<
> https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$
> >
> Toby (Cc'ed) might know it. Or could you provide an example?
>
> --Junchao Zhang
>
>
> On Fri, Apr 25, 2025 at 3:31 AM Klaij, Christiaan via petsc-users <
> petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:
> petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>> wrote:
> We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with
> the nested logging. Any ideas?
>
> Chris
>
>
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: General MPI error
> [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer
> [1]PETSC ERROR: See
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$
> <
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$>
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025
> [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on
> marclus3login2 by jwindt Fri Apr 25 08:52:30 2025
> [1]PETSC ERROR: Configure options:
> --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu
> --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0
> --with-debugging=0 --download-superlu_dist=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$>
> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0
> --download-parmetis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$>
> --download-metis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$>
> --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild
> --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall
> -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall
> -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall
> -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall
> -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG"
> [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330
> [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420
> [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443
> [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405
> [1]PETSC ERROR: #7 PetscLogHandlerView() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342
> [1]PETSC ERROR: #8 PetscLogView() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 98.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [cid:ii_196725d1e2a809852191]
> dr. ir. Christiaan Klaij
> | senior researcher | Research & Development | CFD Development
> T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | c.kl...@marin.nl
> <mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>
> |
> https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$
> <
> https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$
> >
> [Facebook]<
> https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$
> >
> [LinkedIn]<
> https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$
> >
> [YouTube]<
> https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$
> >
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4Go3vEyJ$
>
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4NWbA6Qw$
>
> >
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4Go3vEyJ$
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eq1kivi8yzuD_s9lNwIeqkMg9dU4SC9grCBJnOhoALEg4iILbgybeYt5c__oPKrFt56W5KKM4Kab4NWbA6Qw$
>
