Hi,
I recently upgraded to valgrind 3.11.0 (was 3.8.1 before).
I also upgraded Intel MKL to the 2015 version.
We use valgrind since 2011-10-27 to test our code nightly with about
2200 tests.
Since this upgrade, some tests (not always the same) have the same
trouble (see full log in attachment):
================================
...
blockSane: fail -- redzone-hi
valgrind: m_mallocfree.c:2042 (vgPlain_arena_free): Assertion
'blockSane(a, b)' failed.
host stacktrace:
...
================================
I would like some help to debug this.
It is quite annoying since for the same test, the problem appear or
disappear from one day over another...
Today we got the problem on 6 tests (on 2200), yesterday, only 1 test
failed, before yesterday, none failed....
Thanks for any insights or clues!
Eric
==21183== Memcheck, a memory error detector
==21183== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21183== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21183== Command:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/bin/Test.MortierElasticite.opt
mpi_v=2 verbose=True Beowulf=False outilMassif=False outilVtune=False
outilPerfRecord=False verifValgrind=True Mode=PasCreeEtalon
outilCallgrind=False
RepertoireDestination=/users/cmpbib/compilations/linux_gcc-4.5.1_64bit_valgrind/TV2016-10-19_04h14m48sEDT/opt/Test.MortierElasticite
RepertoireTest=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/TestValidation/Ressources/opt/Test.MortierElasticite
Prefixe=carres_jumeaux_04 outilPerfStat=False
==21183== Parent PID: 21181
==21183==
--21183--
--21183-- Valgrind options:
--21183--
--suppressions=/opt/openmpi-1.8.4/share/openmpi/openmpi-valgrind.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.autres.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.BCS.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.DLOPEN.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.FEAP.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.IOAPI.HDF.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.IOAPI.NATIVE.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.MKL.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.OPENMPI.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.OPENMPI163.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.PETSC.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.PGI.supp
--21183--
--suppressions=/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.VALGRIND.supp
--21183-- -v
--21183-- --gen-suppressions=all
--21183-- --leak-check=full
--21183-- --leak-resolution=high
--21183-- --show-reachable=yes
--21183-- --error-limit=no
--21183-- --num-callers=24
--21183-- --track-fds=yes
--21183--
--log-file=/users/cmpbib/compilations/linux_gcc-4.5.1_64bit_valgrind/TV2016-10-19_04h14m48sEDT/opt/Test.MortierElasticite/valgrind_out.carres_jumeaux_04.n%q{OMPI_COMM_WORLD_RANK}
--21183-- --track-origins=yes
--21183-- Contents of /proc/version:
--21183-- Linux version 3.7.10-1.45-desktop (geeko@buildhost) (gcc version
4.7.2 20130108 [gcc-4_7-branch revision 195012] (SUSE Linux) ) #1 SMP PREEMPT
Tue Dec 16 20:27:58 UTC 2014 (4c885a1)
--21183--
--21183-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-rdtscp-sse3
--21183-- Page sizes: currently 4096, max supported 4096
--21183-- Valgrind library directory: /usr/local/valgrind-3.11.0/lib64/valgrind
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/bin/Test.MortierElasticite.opt
--21183-- Reading syms from /lib64/ld-2.17.so
--21183-- Reading syms from
/usr/local/valgrind-3.11.0/lib64/valgrind/memcheck-amd64-linux
--21183-- object doesn't have a dynamic symbol table
--21183-- Scheduler: using generic scheduler lock implementation.
--21183-- Reading suppressions file:
/opt/openmpi-1.8.4/share/openmpi/openmpi-valgrind.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.autres.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.BCS.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.DLOPEN.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.FEAP.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.IOAPI.HDF.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.IOAPI.NATIVE.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.MKL.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.OPENMPI.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.OPENMPI163.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.PETSC.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.PGI.supp
--21183-- Reading suppressions file:
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/scripts/valgrind/valgrind.VALGRIND.supp
--21183-- Reading suppressions file:
/usr/local/valgrind-3.11.0/lib64/valgrind/default.supp
==21183== embedded gdbserver: reading from
/tmp/vgdb-pipe-from-vgdb-to-21183-by-???-on-???
==21183== embedded gdbserver: writing to
/tmp/vgdb-pipe-to-vgdb-from-21183-by-???-on-???
==21183== embedded gdbserver: shared mem
/tmp/vgdb-pipe-shared-mem-vgdb-21183-by-???-on-???
==21183==
==21183== TO CONTROL THIS PROCESS USING vgdb (which you probably
==21183== don't want to do, unless you know exactly what you're doing,
==21183== or are doing some strange experiment):
==21183== /usr/local/valgrind-3.11.0/lib64/valgrind/../../bin/vgdb
--pid=21183 ...command...
==21183==
==21183== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==21183== /path/to/gdb
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/bin/Test.MortierElasticite.opt
==21183== and then give GDB the following command
==21183== target remote |
/usr/local/valgrind-3.11.0/lib64/valgrind/../../bin/vgdb --pid=21183
==21183== --pid is optional if only one valgrind process is running
==21183==
--21183-- REDIR: 0x40177a0 (ld-linux-x86-64.so.2:strlen) redirected to
0x38057531 (vgPlain_amd64_linux_REDIR_FOR_strlen)
--21183-- Reading syms from
/usr/local/valgrind-3.11.0/lib64/valgrind/vgpreload_core-amd64-linux.so
--21183-- Reading syms from
/usr/local/valgrind-3.11.0/lib64/valgrind/vgpreload_memcheck-amd64-linux.so
==21183== WARNING: new redirection conflicts with existing -- ignoring it
--21183-- old: 0x040177a0 (strlen ) R-> (0000.0) 0x38057531
vgPlain_amd64_linux_REDIR_FOR_strlen
--21183-- new: 0x040177a0 (strlen ) R-> (2007.0) 0x04c2d170
strlen
--21183-- REDIR: 0x4017610 (ld-linux-x86-64.so.2:index) redirected to 0x4c2cd50
(index)
--21183-- REDIR: 0x4017690 (ld-linux-x86-64.so.2:strcmp) redirected to
0x4c2e220 (strcmp)
--21183-- REDIR: 0x40187c0 (ld-linux-x86-64.so.2:mempcpy) redirected to
0x4c30fb0 (mempcpy)
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Adaptation.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ConditionsLimites.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_MVGlobal.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsUtil.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_LecteurDeclaration.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsScalaire.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsV3D.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsV2D.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsTensO2Sym.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsTensO2NSym.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ChampsTensO4Sym.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_ExpressionAlgebrique.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Coquilles.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_PtIntegration.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Champs.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Interpolation.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Geometrie.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Elements.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Formulation.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_TFGeneral.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_TFormulation.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_TFMecSolides.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_TFMasseGeneralise.so
--21183-- Reading syms from
/pmi/cmpbib/compilation_BIB_gcc-4.5.1_64bit_valgrind/COMPILE_AUTO/GIREF/lib/libgiref_opt_Interface.so
blockSane: fail -- redzone-hi
valgrind: m_mallocfree.c:2042 (vgPlain_arena_free): Assertion 'blockSane(a, b)'
failed.
host stacktrace:
==21183== at 0x3803D4C8: show_sched_status_wrk (m_libcassert.c:343)
==21183== by 0x3803D5D4: report_and_quit (m_libcassert.c:415)
==21183== by 0x3803D761: vgPlain_assert_fail (m_libcassert.c:481)
==21183== by 0x3804BD3C: vgPlain_arena_free (m_mallocfree.c:2042)
==21183== by 0x380D611E: vgModuleLocal_img_done (image.c:778)
==21183== by 0x38073E84: vgModuleLocal_read_elf_debug_info (readelf.c:3027)
==21183== by 0x3806C4C9: di_notify_ACHIEVE_ACCEPT_STATE (debuginfo.c:749)
==21183== by 0x3806C4C9: vgPlain_di_notify_mmap (debuginfo.c:1067)
==21183== by 0x3809391C: vgModuleLocal_generic_PRE_sys_mmap
(syswrap-generic.c:2367)
==21183== by 0x380C6F4A: vgSysWrap_amd64_linux_sys_mmap_before
(syswrap-amd64-linux.c:637)
==21183== by 0x3808FEA1: vgPlain_client_syscall (syswrap-main.c:1905)
==21183== by 0x3808CA92: handle_syscall (scheduler.c:1118)
==21183== by 0x3808E566: vgPlain_scheduler (scheduler.c:1435)
==21183== by 0x3809DFA3: thread_wrapper (syswrap-linux.c:102)
==21183== by 0x3809DFA3: run_a_thread_NORETURN (syswrap-linux.c:155)
sched status:
running_tid=1
Thread 1: status = VgTs_Runnable (lwpid 21183)
==21183== at 0x401721A: mmap (in /lib64/ld-2.17.so)
==21183== by 0x4006877: _dl_map_object_from_fd (in /lib64/ld-2.17.so)
==21183== by 0x400823C: _dl_map_object (in /lib64/ld-2.17.so)
==21183== by 0x400C2F1: openaux (in /lib64/ld-2.17.so)
==21183== by 0x400E7A5: _dl_catch_error (in /lib64/ld-2.17.so)
==21183== by 0x400C523: _dl_map_object_deps (in /lib64/ld-2.17.so)
==21183== by 0x4003543: dl_main (in /lib64/ld-2.17.so)
==21183== by 0x40150BD: _dl_sysdep_start (in /lib64/ld-2.17.so)
==21183== by 0x4004D85: _dl_start (in /lib64/ld-2.17.so)
==21183== by 0x4001527: ??? (in /lib64/ld-2.17.so)
==21183== by 0xD: ???
==21183== by 0xFFEFFE1CA: ???
==21183== by 0xFFEFFE231: ???
==21183== by 0xFFEFFE239: ???
==21183== by 0xFFEFFE246: ???
==21183== by 0xFFEFFE254: ???
==21183== by 0xFFEFFE266: ???
==21183== by 0xFFEFFE277: ???
==21183== by 0xFFEFFE28D: ???
==21183== by 0xFFEFFE2A0: ???
==21183== by 0xFFEFFE2B3: ???
==21183== by 0xFFEFFE2C8: ???
==21183== by 0xFFEFFE34D: ???
==21183== by 0xFFEFFE3D3: ???
==21183== by 0xFFEFFE3ED: ???
Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.
If that doesn't help, please report this bug to: www.valgrind.org
In the bug report, send all the above text, the valgrind
version, and what OS and version you are using. Thanks.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Valgrind-users mailing list
Valgrind-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/valgrind-users
