---
** [tickets:#2304] imm: osafimmpbed creates coredump due to double free memory**
**Status:** accepted
**Milestone:** 5.2.FC
**Created:** Mon Feb 13, 2017 11:57 AM UTC by Zoran Milinkovic
**Last Updated:** Mon Feb 13, 2017 11:57 AM UTC
**Owner:** Zoran Milinkovic
When IMM is running with code coverage, there is often coredump for osafimmpbed.
The problem comes from double exit call from two threads, the main and MDS
thread. Both threads try to call destructor for static variable in IMM PBE
library.
I think this is a timing issue and we haven't seen this error earlier. With
code coverage flag, the problem occurs aprox. once a day.
GDB coredump backtrace:
~~~
[New LWP 1888]
[New LWP 1884]
[New LWP 1887]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/lib/opensaf/osafimmpbed --pbe
/srv/shared/imm//imm.db'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fc923fbcc37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
Thread 3 (Thread 0x7fc9258e8b00 (LWP 1887)):
#0 0x00007fc924072fdd in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007fc924ad909b in osaf_poll_no_timeout (io_fds=0x7fc9258e8290,
i_nfds=1) at src/base/osaf_poll.c:32
result = 32713
#2 0x00007fc924ad9248 in osaf_ppoll (io_fds=0x7fc9258e8290, i_nfds=1,
i_timeout_ts=0x0, i_sigmask=0x0) at src/base/osaf_poll.c:79
millisecond_round_up = {tv_sec = 0, tv_nsec = 999999}
max_possible_timeout = {tv_sec = 2147483, tv_nsec = 647000000}
start_time = {tv_sec = 17179869186, tv_nsec = 140501895252736}
time_left_ts = {tv_sec = 1, tv_nsec = 1}
result = 615339859
#3 0x00007fc924ae95cf in ncs_tmr_wait () at src/base/sysf_tmr.c:409
rc = 1
inds_rmvd = 1
next_delay = 0
tv = {tv_sec = 16777215, tv_usec = 0}
ts_current = {tv_sec = 216961, tv_nsec = 620030550}
ts = {tv_sec = 16777215, tv_nsec = 0}
set = {fd = 8, events = 1, revents = 0}
#4 0x00007fc924353184 in start_thread (arg=0x7fc9258e8b00) at
pthread_create.c:312
__res = <optimized out>
pd = 0x7fc9258e8b00
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140501895252736,
-8535808571625374835, 1, 1, 140501895253440, 140501895252736,
8509828887122344845, 8509832138929142669}, mask_was_saved = 0}}, priv = {pad =
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#5 0x00007fc92408037d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Thread 2 (Thread 0x7fc9258eb780 (LWP 1884)):
#0 0x00007fc92435a64a in do_fcntl (arg=0x7ffd194a7070, cmd=7, fd=22) at
../sysdeps/unix/sysv/linux/fcntl.c:39
resultvar = 18446744073709551104
#1 __libc_fcntl (fd=22, cmd=<optimized out>) at
../sysdeps/unix/sysv/linux/fcntl.c:92
ap = {{gp_offset = 16, fp_offset = 32713, overflow_arg_area =
0x7ffd194a7070, reg_save_area = 0x7ffd194a7030}}
arg = 0x7ffd194a7070
oldtype = 0
#2 0x00007fc925270985 in __gcov_open () from
/usr/local/lib/opensaf/libosaf_common.so.0
No symbol table info available.
#3 0x00007fc9252714ee in gcov_exit () from
/usr/local/lib/opensaf/libosaf_common.so.0
No symbol table info available.
#4 0x00007fc923fc21a9 in __run_exit_handlers (status=1, listp=0x7fc9243446c8
<__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
atfct = <optimized out>
onfct = <optimized out>
cxafct = <optimized out>
f = <optimized out>
#5 0x00007fc923fc21f5 in __GI_exit (status=<optimized out>) at exit.c:104
No locals.
#6 0x000055828aa7c60c in pbeDaemon (immHandle=4230542917903,
dbHandle=0x55828bb010e8, ownerHandle=1483565869334821379,
classIdMap=0x7ffd194abc10, objCount=335, pbe2=false, pbe2B=false) at
src/imm/immpbed/immpbe_daemon.cc:2343
error = SA_AIS_OK
ci = {first = <error reading variable: Cannot access memory at address
0x26>, second = }
__FUNCTION__ = "pbeDaemon"
#7 0x000055828aa6b408 in main (argc=3, argv=0x7ffd194abdd8) at
src/imm/immpbed/immpbe.cc:354
localTmpFilename = ""
pbeRecoverFile = true
dbHandle = 0x55828bb010e8
classIdMap = std::map with 62 elements = {["OpenSafLogConfig"] =
0x55828bb83290, ["OpenSafLogCurrentConfig"] = 0x55828bb792a0,
["OpenSafSmfCampRestartIndicator"] = 0x55828bb7cdb0,
["OpenSafSmfCampRestartInfo"] = 0x55828bb83b80, ["OpenSafSmfConfig"] =
0x55828bb79090, ["OpenSafSmfExecControl"] = 0x55828bb7d0d0, ["OpenSafSmfMisc"]
= 0x55828bb7bbc0, ["OpenSafSmfPbeIndicator"] = 0x55828bb79420,
["OpenSafSmfRollbackData"] = 0x55828bb7a650, ["OpenSafSmfRollbackElement"] =
0x55828bb7afd0, ["OpenSafSmfSingleStepInfo"] = 0x55828bb79b50,
["OpensafConfig"] = 0x55828bb782c0, ["OpensafImm"] = 0x55828bb86a10,
["OsafImmPbeRt"] = 0x55828bb78330, ["SaAmfAppBaseType"] = 0x55828bb71e40,
["SaAmfAppType"] = 0x55828bb76070, ["SaAmfApplication"] = 0x55828bb6ae10,
["SaAmfCSBaseType"] = 0x55828bb6ad60, ["SaAmfCSI"] = 0x55828bb74520,
["SaAmfCSIAssignment"] = 0x55828bb66e90, ["SaAmfCSIAttribute"] =
0x55828bb71230, ["SaAmfCSType"] = 0x55828bb73760, ["SaAmfCluster"] =
0x55828bb6d2b0, ["SaAmfComp"] = 0x558
28bb6ba70, ["SaAmfCompBaseType"] = 0x55828bb6db80, ["SaAmfCompCsType"] =
0x55828bb67190, ["SaAmfCompGlobalAttributes"] = 0x55828bb6d9a0,
["SaAmfCompType"] = 0x55828bb670d0, ["SaAmfCtCsType"] = 0x55828bb67d90,
["SaAmfHealthcheck"] = 0x55828bb63290, ["SaAmfHealthcheckType"] =
0x55828bb63180, ["SaAmfNode"] = 0x55828bb648a0, ["SaAmfNodeGroup"] =
0x55828bb64d10, ["SaAmfNodeSwBundle"] = 0x55828bb64f80, ["SaAmfSG"] =
0x55828bb5fb30, ["SaAmfSGBaseType"] = 0x55828bb61660, ["SaAmfSGType"] =
0x55828bb60560, ["SaAmfSI"] = 0x55828bb5ac50, ["SaAmfSIAssignment"] =
0x55828bb57aa0, ["SaAmfSIDependency"] = 0x55828bb59a40, ["SaAmfSIRankedSU"] =
0x55828bb5ab30, ["SaAmfSU"] = 0x55828bb58850, ["SaAmfSUBaseType"] =
0x55828bb57bc0, ["SaAmfSUType"] = 0x55828bb55df0, ["SaAmfSutCompType"] =
0x55828bb56050, ["SaAmfSvcBaseType"] = 0x55828bb55a10, ["SaAmfSvcType"] =
0x55828bb53b80, ["SaAmfSvcTypeCSTypes"] = 0x55828bb54820, ["SaCkptCheckpoint"]
= 0x55828bb4ff70, ["SaCkptReplica"] = 0x55828bb54210, ["SaClmCluster"
] = 0x55828bb500b0, ["SaClmNode"] = 0x55828bb4e340, ["SaImmMngt"] =
0x55828bb515a0, ["SaLogStream"] = 0x55828bb50cb0, ["SaLogStreamConfig"] =
0x55828bb4be20, ["SaSmfActivationUnit"] = 0x55828bb4e910, ["SaSmfCampaign"] =
0x55828bb47b50, ["SaSmfDeactivationUnit"] = 0x55828bb49720, ["SaSmfImageNodes"]
= 0x55828bb482d0, ["SaSmfProcedure"] = 0x55828bb49ad0, ["SaSmfStep"] =
0x55828bb47870, ["SaSmfSwBundle"] = 0x55828bb43600}
objCount = 335
logPath = 0x55828aa83ce5 "osafimmpbed_trace"
pbe2 = false
fileReOpened = true
errorCode = SA_AIS_OK
admoRetVal = SA_AIS_OK
pbeDumpCase = false
i = 3
c = -1
immHandle = 4230542917903
version = {releaseCode = 65 'A', majorVersion = 2 '\002', minorVersion
= 17 '\021'}
ownerHandle = 1483565869334821379
classRDNMap = std::map with 0 elements
category_mask = 0
trace_label = 0x55828aa83cf7 "osafimmpbed"
retryInterval = 1000000
maxTries = 70
filename = "/srv/shared/imm//imm.db"
defaultLog = 0x55828aa83ce5 "osafimmpbed_trace"
tryCount = 1
params = {0x0}
__PRETTY_FUNCTION__ = "int main(int, char**)"
long_options = {{name = 0x55828aa840b3 "recover", has_arg = 0, flag =
0x0, val = 114}, {name = 0x55828aa840bb "pbe", has_arg = 1, flag = 0x0, val =
112}, {name = 0x55828aa840bf "pbe2A", has_arg = 1, flag = 0x0, val = 65}, {name
= 0x55828aa840c5 "pbe2B", has_arg = 1, flag = 0x0, val = 66}, {name = 0x0,
has_arg = 0, flag = 0x0, val = 0}}
pbe2BCase = false
dump_trace_label = 0x55828aa83cf7 "osafimmpbed"
retParams = 0x0
Thread 1 (Thread 0x7fc9258c8b00 (LWP 1888)):
#0 0x00007fc923fbcc37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 1884
selftid = 1888
#1 0x00007fc923fc0028 in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0},
sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007fc923ff92a4 in __libc_message (do_abort=do_abort@entry=1,
fmt=fmt@entry=0x7fc9241076b0 "*** Error in `%s': %s: 0x%s ***\n") at
../sysdeps/posix/libc_fatal.c:175
ap = {{gp_offset = 40, fp_offset = 0, overflow_arg_area =
0x7fc9258c7bc0, reg_save_area = 0x7fc9258c7b50}}
fd = 2
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
#3 0x00007fc92400555e in malloc_printerr (ptr=<optimized out>,
str=0x7fc924107878 "double free or corruption (fasttop)", action=1) at
malloc.c:4996
buf = "000055828bb15eb0"
cp = <optimized out>
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at
malloc.c:3840
size = <optimized out>
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
errstr = <optimized out>
locked = <optimized out>
#5 0x00007fc92483936f in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::~basic_string() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6 0x00007fc923fc253a in __cxa_finalize (d=0x7fc9256c6c80) at cxa_finalize.c:56
check = 15
cxafn = <optimized out>
cxaarg = <optimized out>
f = 0x7fc924345fd0 <initial+336>
funcs = 0x7fc924345e80 <initial>
#7 0x00007fc925493833 in __do_global_dtors_aux () from
/usr/local/lib/opensaf/libimmpbe_dump.so.0
No symbol table info available.
#8 0x00007fc9258c7dc0 in ?? ()
No symbol table info available.
#9 0x00007fc9256e870a in _dl_fini () at dl-fini.c:252
array = 0x7fc9256c5d78
i = 0
nmaps = 32713
nloaded = <optimized out>
i = 2
l = 0x7fc9258f95d0
ns = 140501868881210
maps = 0x7fc9258c7cc0
maps_size = 140501893016704
do_audit = 630166992
__PRETTY_FUNCTION__ = "_dl_fini"
Backtrace stopped: frame did not save the PC
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
~~~
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
