If I deactivate all the calls to PetscLogEventBegin/End in the simulation code, the error does not show-up.
But since there are more than 2500 events, it's impossible to do them one-by-one, especially since the error shows-up at random and requires a number of cases and repetitions. Unfortunately, I'm running out of time and budget. I will retry once we get Rocky Linux 9 and the latest Intel compilers. Chris ________________________________________ From: Stefano Zampini <stefano.zamp...@gmail.com> Sent: Thursday, May 1, 2025 10:57 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Il giorno gio 1 mag 2025 alle ore 11:38 Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> ha scritto: The checks seem to be in place: I do get a PETSC ERROR when I add a log event on rank 0 as you suggested. Ok, the broken logic may be in LogView then. You can try deactivating some logging by classes and see how the error evolves, maybe using PetscLogClassSetActiveAll. Or, if feasible, commenting out some part of the simulation code Another thought: in between the log events pairs, I also have calls to PetscLogFlops, perhaps that plays a role somehow. It shouldn't Chris ________________________________________ From: Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl>> Sent: Thursday, May 1, 2025 10:23 AM To: Stefano Zampini Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Was the rewritting by Toby done somewhere between petsc 3.19.4 (no problem) and 3.23.4 (problem)? Chris ________________________________________ From: Stefano Zampini <stefano.zamp...@gmail.com<mailto:stefano.zamp...@gmail.com>> Sent: Thursday, May 1, 2025 9:12 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging If I look at the code for PetscLogHandlerEventBegin_Default, there are checks in place to see if the event is collectively called (see below) Can you make sure the Nested logging system has the same checks? It should, but double check since the code has been largely rewritten by Toby some time ago; to check it should be as easy as writing a code that calls a collective event on a single process and a debug version of petsc should complain if (rank ==0) LogEventBegin() <-this should call MPIU_Allreduce, but other processes will not, thus hang If it does not complain, then the error must come from how the logic in LogView works, and from how it traverses the various events (my guess: processed in a different order from different processes). Without a reproducer, it is hard to understand what's going on static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, PetscObject o4) { PetscLogHandler_Default def = (PetscLogHandler_Default)h->data; PetscEventPerfInfo *event_perf_info = NULL; PetscLogEventInfo event_info; PetscLogDouble time; PetscLogState state; PetscLogStage stage; PetscFunctionBegin; PetscCall(PetscLogHandlerGetState(h, &state)); if (PetscDefined(USE_DEBUG)) { PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); if (event_info.collective && o1) { PetscInt64 b1[2], b2[2]; b1[0] = -o1->cidx; b1[1] = o1->cidx; PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, PetscObjectComm(o1))); PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, "Collective event %s not called collectively %" PetscInt64_FMT " != %" PetscInt64_FMT, event_info.name<https://urldefense.us/v3/__http://event_info.name__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYGuEWpiE$ ><https://urldefense.us/v3/__http://event_info.name__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYGuEWpiE$ >, -b2[0], b2[1]); } } /* Synchronization */ PetscCall(PetscLogHandlerEventSync_Default(h, event, PetscObjectComm(o1))); Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>> ha scritto: I've tried with -log_sync, no complaints whatsoever, but the error is still there... Chris ________________________________________ From: Stefano Zampini <stefano.zamp...@gmail.com<mailto:stefano.zamp...@gmail.com><mailto:stefano.zamp...@gmail.com<mailto:stefano.zamp...@gmail.com>>> Sent: Tuesday, April 29, 2025 6:12 PM To: Randall Mackie Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called Stefano On Tue, Apr 29, 2025, 18:21 Randall Mackie <rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>>>> wrote: ah okay, I missed that this was found using openmpi. then it’s probably not the same issue we had. I can’t remember in which version it was fixed (I’m away from my work computer)….I do know in our case openmpi and the latest Intel One API work fine. Randy On Apr 29, 2025, at 8:58 AM, Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>> wrote: Well, the error below only shows-up thanks to openmpi and gnu compilers. With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed? Chris ________________________________________ <image374583.png> dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYrepzcmY$ <https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYrepzcmY$ ><https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYrepzcmY$ ><https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cx-ci9iNg$> <image322757.png><https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cxl-rBbfA$> <image512282.png><https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cw7FNelIQ$> <image090879.png><https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwmfHQRog$> From: Randall Mackie <rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>>>> Sent: Tuesday, April 29, 2025 3:33 PM To: Klaij, Christiaan Cc: Matthew Knepley; petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com><mailto:rlmackie...@gmail.com<mailto:rlmackie...@gmail.com>>>. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYW5_HLdI$ <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwgJDUS5w$>> We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using. Can you try a different MPI or the latest Intel One API and see if your error clears? Randy On Tue, Apr 29, 2025 at 8:17 AM Klaij, Christiaan via petsc-users <petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>> wrote: I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message. Chris ________________________________________ From: Matthew Knepley <knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>>>> Sent: Tuesday, April 29, 2025 1:50 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging On Tue, Apr 29, 2025 at 6:50 AM Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>>> wrote: Here's a slightly better error message, obtained --with-debugging=1 Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 [0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1 [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 [0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 ________________________________________ [cid:ii_19681617e7812ff9cfc1] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$ > [Facebook]<https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$ > [LinkedIn]<https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$ > [YouTube]<https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$ > From: Klaij, Christiaan <c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>>> Sent: Monday, April 28, 2025 3:53 PM To: Matthew Knepley Cc: Junchao Zhang; petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components. Chris ________________________________________ From: Matthew Knepley <knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>>>>> Sent: Monday, April 28, 2025 3:06 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>><mailto:knep...@gmail.com<mailto:knep...@gmail.com><mailto:knep...@gmail.com<mailto:knep...@gmail.com>>>>>. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$ > On Mon, Apr 28, 2025 at 8:45 AM Klaij, Christiaan via petsc-users <petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>>> wrote: I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, but it does not replicate the problem and works fine. Perhaps it is related to fortran, since the manualpage of PetscLogNestedBegin says "No fortran support" (why? we've been using it in fortran ever since). Therefore I've tried adding it to src/snes/ex5f90.F90 and that also works fine. It seems I cannot replicate the problem in a small example, unfortunately. We cannot replicate it here. Is there a chance you could bisect to see what change is responsible? Thanks, Matt Chris ________________________________________ From: Junchao Zhang <junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>>>>> Sent: Saturday, April 26, 2025 3:51 PM To: Klaij, Christiaan Cc: petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com><mailto:junchao.zh...@gmail.com<mailto:junchao.zh...@gmail.com>>>>>>. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$ > Toby (Cc'ed) might know it. Or could you provide an example? --Junchao Zhang On Fri, Apr 25, 2025 at 3:31 AM Klaij, Christiaan via petsc-users <petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov><mailto:petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>>>>>>> wrote: We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas? Chris [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: General MPI error [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer [1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$> for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 [1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ <https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ <https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$> --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ <https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$> --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 98. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [cid:ii_196725d1e2a809852191] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl><mailto:c.kl...@marin.nl<mailto:c.kl...@marin.nl>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$> [Facebook]<https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$> [LinkedIn]<https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$> [YouTube]<https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > -- Stefano -- Stefano
