Daniel, Thanks very much! I received the binary and we’ll figure this out.
Best, John -- John Mellor-Crummey Professor Dept of Computer Science Rice University email: joh...@rice.edu phone: 713-348-5179 > On May 12, 2025, at 10:05 AM, Daniel Kokron - NOAA Affiliate > <daniel.kok...@noaa.gov> wrote: > > Got permission to share the executable. Link sent. > > On Fri, May 9, 2025 at 2:26 PM Daniel Kokron - NOAA Affiliate > <daniel.kok...@noaa.gov <mailto:daniel.kok...@noaa.gov>> wrote: >> I'll ask about providing the executable. >> >> On Fri, May 9, 2025 at 1:52 PM John Mellor-Crummey <joh...@rice.edu >> <mailto:joh...@rice.edu>> wrote: >>> Hi Daniel, >>> >>> Thanks for the callstack. >>> >>> The problem seems to be exactly the same one recently encountered by Doug >>> Pase for a Fortran program at Sandia. This is a problem inside the type >>> processing by the Dyninst software written by our collaborators. >>> >>> Can you share a binary with us to facilitate debugging? The Sandia binary >>> is export controlled and only accessible inside their firewall. Having a >>> non-export controlled binary for debugging would make our lives easier. >>> >>> Best, >>> >>> John >>> -- >>> John Mellor-Crummey Professor >>> Dept of Computer Science Rice University >>> email: joh...@rice.edu <mailto:joh...@rice.edu> phone: 713-348-5179 >>> >>> >>> >>>> On May 9, 2025, at 12:15 PM, Daniel Kokron - NOAA Affiliate >>>> <daniel.kok...@noaa.gov <mailto:daniel.kok...@noaa.gov>> wrote: >>>> >>>> The application is compiled with Intel ifort. HPCToolkit and its >>>> dependencies are compiled with gcc-13.2.1. I attached the spec for >>>> HPCToolkit. >>>> >>>> >>>> (gdb) run --nocache >>>> /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x >>>> Starting program: >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct >>>> --nocache >>>> /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x >>>> Missing separate debuginfos, use: zypper install >>>> glibc-debuginfo-2.31-150300.63.1.x86_64 >>>> Missing separate debuginfo for >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libstdc++.so.6 >>>> Try: zypper install -C >>>> "debuginfo(build-id)=c74eca671e2dd0f063706372d103f8acef88f1e3" >>>> Missing separate debuginfo for >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1 >>>> Try: zypper install -C >>>> "debuginfo(build-id)=54684492738e640bcd600e830cee025dd8771a20" >>>> Missing separate debuginfo for >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgcc_s.so.1 >>>> Try: zypper install -C >>>> "debuginfo(build-id)=12f775ec4aeb94b749897b1b65638f18b61d1b1f" >>>> [Thread debugging using libthread_db enabled] >>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>> begin sequential analysis of CPU binary >>>> /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x >>>> (size = 377978672, threads = 1) >>>> hpcstruct: >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/boost-1.87.0-2cldxfpwec5rbbhxutja5lwcgzh6fbhc/include/boost/smart_ptr/shared_ptr.hpp:550: >>>> typename boost::detail::sp_member_access<T>::type >>>> boost::shared_ptr<T>::operator->() const [with T = >>>> Dyninst::SymtabAPI::typeSubrange; typename >>>> boost::detail::sp_member_access<T>::type = >>>> Dyninst::SymtabAPI::typeSubrange*]: Assertion `px != 0' failed. >>>> >>>> Program received signal SIGABRT, Aborted. >>>> 0x0000155553e2fd2b in raise () from /lib64/libc.so.6 >>>> (gdb) where >>>> #0 0x0000155553e2fd2b in raise () from /lib64/libc.so.6 >>>> #1 0x0000155553e313e5 in abort () from /lib64/libc.so.6 >>>> #2 0x0000155553e27c6a in __assert_fail_base () from /lib64/libc.so.6 >>>> #3 0x0000155553e27cf2 in __assert_fail () from /lib64/libc.so.6 >>>> #4 0x0000155554d65127 in boost::enable_if<boost::integral_constant<bool, >>>> !((bool)boost::is_same<Dyninst::SymtabAPI::Type, >>>> Dyninst::SymtabAPI::typeSubrange>::value)>, >>>> boost::shared_ptr<Dyninst::SymtabAPI::Type> >::type >>>> Dyninst::SymtabAPI::typeCollection::addOrUpdateType<Dyninst::SymtabAPI::typeSubrange>(boost::shared_ptr<Dyninst::SymtabAPI::typeSubrange>) >>>> () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #5 0x0000155554d547e6 in Dyninst::SymtabAPI::DwarfWalker::parseSubrange() >>>> () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #6 0x0000155554d5a0a8 in >>>> Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #7 0x0000155554d5b235 in >>>> Dyninst::SymtabAPI::DwarfWalker::findAnyType(Dwarf_Attribute, bool, >>>> boost::shared_ptr<Dyninst::SymtabAPI::Type>&) () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #8 0x0000155554d5b732 in >>>> Dyninst::SymtabAPI::DwarfWalker::findType(boost::shared_ptr<Dyninst::SymtabAPI::Type>&, >>>> bool) () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #9 0x0000155554d5497b in Dyninst::SymtabAPI::DwarfWalker::parseArray() () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #10 0x0000155554d59fb8 in >>>> Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #11 0x0000155554d5a53b in >>>> Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #12 0x0000155554d5bab0 in >>>> Dyninst::SymtabAPI::DwarfWalker::parseModule(Dwarf_Die, >>>> Dyninst::SymtabAPI::Module*&) [clone .constprop.0] [clone .isra.0] () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #13 0x0000155554d5c15c in Dyninst::SymtabAPI::DwarfWalker::parse() [clone >>>> ._omp_fn.0] () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #14 0x000015555403b306 in GOMP_parallel () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1 >>>> #15 0x0000155554d5d2ed in Dyninst::SymtabAPI::DwarfWalker::parse() () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #16 0x0000155554d0a0c1 in Dyninst::SymtabAPI::Object::parseTypeInfo() () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #17 0x0000155554cd48a7 in Dyninst::SymtabAPI::Symtab::parseTypes() () from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #18 0x0000155553fef5d7 in __pthread_once_slow () from >>>> /lib64/libpthread.so.0 >>>> #19 0x0000155554ccc8b4 in Dyninst::SymtabAPI::Symtab::parseTypesNow() () >>>> from >>>> /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 >>>> #20 0x00000000004418c8 in Inline::openSymtab >>>> (elfFile=elfFile@entry=0x8d94b0) at Struct-Inline.cpp:132 >>>> #21 0x000000000043cb31 in BAnal::Struct::makeStructure (filename=..., >>>> outFile=outFile@entry=0x830ab0, gapsFile=gapsFile@entry=0x0, >>>> gaps_filenm=..., search_path=..., structOpts=...) at Struct.cpp:770 >>>> #22 0x000000000042cb42 in doSingleBinary (args=..., >>>> sb=sb@entry=0x7ffffffd8740) at >>>> /usr/include/c++/13/bits/basic_string.tcc:238 >>>> #23 0x0000000000412cfd in realmain (argc=<optimized out>, argv=<optimized >>>> out>) at main.cpp:209 >>>> #24 0x000000000041220a in main (argc=<optimized out>, argv=<optimized >>>> out>) at main.cpp:137 >>>> >>>> On Fri, May 9, 2025 at 11:16 AM John Mellor-Crummey <joh...@rice.edu >>>> <mailto:joh...@rice.edu>> wrote: >>>>> Hi Daniel, >>>>> >>>>> You should be able to run hpcstruct under gdb and then run it directly on >>>>> the offending binary as follows >>>>> >>>>> gdb `which hpcstruct` >>>>> run --nocache /path/to/gfs_model >>>>> >>>>> Then, you can send us a call path. By any chance is this a Fortran code >>>>> compiled with gfortran? We are presently looking into a complaint about >>>>> that from Sandia. >>>>> >>>>> Best, >>>>> >>>>> John >>>>> -- >>>>> John Mellor-Crummey Professor >>>>> Dept of Computer Science Rice University >>>>> email: joh...@rice.edu <mailto:joh...@rice.edu> phone: 713-348-5179 >>>>> >>>>> >>>>> >>>>>> On May 9, 2025, at 8:59 AM, Daniel Kokron - NOAA Affiliate via >>>>>> HPCToolkit-forum <hpctoolkit-fo...@mailman.rice.edu >>>>>> <mailto:hpctoolkit-fo...@mailman.rice.edu>> wrote: >>>>>> >>>>>> I am encountering the following error while running hpcstruct. I cannot >>>>>> find the core file in any of the usual places. I have also tried >>>>>> running hpcstruct under gdb without getting very far. >>>>>> >>>>>> Wondering what my debugging options are? >>>>>> >>>>>> begin concurrent analysis of CPU binary gfs_model. (size = 377978416, >>>>>> threads = 1) >>>>>> /bin/sh: line 32: 63480 Aborted (core dumped) >>>>>> /spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct. >>>>>> --nocache -j 1 -o $struct_name -M $meas_dir >>>>>> /Baseline_6Hr_WithWW3Restarts_Trace.16774.rawdata/cpubins/model.x > >>>>>> $warn_name 2>&1 >>>>>> >>>>>> Dan >>>>>> _______________________________________________ >>>>>> HPCToolkit-forum mailing list >>>>>> hpctoolkit-fo...@mailman.rice.edu >>>>>> <mailto:hpctoolkit-fo...@mailman.rice.edu> >>>>>> https://mailman.rice.edu/mailman/listinfo/hpctoolkit-forum >>>>> >>> >>>
_______________________________________________ Dyninst-api mailing list Dyninst-api@cs.wisc.edu https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
