Hello, The value of debug level I had during the crash is 2. --- debug=2 ---
I checked from my monitoring tools and system logs if the server has encounter any issue (freeze, network lost, database issues, ...) but I found nothing. [image: image.png] [image: image.png] I use Debian (8.6 default version). [image: image.png] Regards Abdoul Le lun. 25 févr. 2019 à 18:34, Daniel-Constantin Mierla <mico...@gmail.com> a écrit : > Hello, > > that's strange, but a while ago someone else reported an issue with same > backtrace. > > So the crash happens at the last line in the next snippet from > reply_received() function in the tm module: > > uac=&t->uac[branch]; > LM_DBG("org. status uas=%d, uac[%d]=%d local=%d is_invite=%d)\n", > t->uas.status, branch, uac->last_received, > is_local(t), is_invite(t)); > last_uac_status=uac->last_received; > > The backtrace and info locals say that uac is null (0x0). According to my > knowledge, the address of a field in a structure cannot be null and uac is > set to &t->uac[branch]. Moreover, uac->last_received is printed in the > LM_DBG() above the line of crash, if uac was 0x0, the crash should have > happened there. > > Then uac is a local variable, so it is on the stack of the process, in its > private memory. There is no other assign or copy operation between the line > of code where the uac is set and the crash. So overall, should be no race > condition there. Either the kernel was doing something wrong, or maybe the > coredump was somehow corrupted. > > What was the value of debug level you had during the crash (debug > parameter in kamailio.cfg)? > > Could there have been any freeze of the operating system for long time and > then a resume? > > Can you give the output of command: > > uname -a > > What kind of linux distro and version you are running? > > Cheers, > Daniel > On 25.02.19 15:27, Abdoul Osséni wrote: > > Hello, > > Please see attached the output of the gdb commands. > > Can you check with all core files and see if the backtrace is the same? > > --> Yes the backtrace is the same. > > Sorry, I use kamailio v5.2 > > root@sbc:/var/cores# kamailio -V > version: kamailio 5.2.1 (x86_64/linux) cd2583 > flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, > Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, > FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, > USE_DST_BLACKLIST, HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, > BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: cd2583 > compiled on 07:33:25 Jan 31 2019 with gcc 4.9.2 > > > Kamailio is running on a bare metal server. > > > Thanks > > Abdoul > > > Le lun. 25 févr. 2019 à 14:40, Daniel-Constantin Mierla <mico...@gmail.com> > a écrit : > >> Hello, >> >> can you give the output for next gdb commands: >> >> bt full >> >> info locals >> >> list >> >> Can you check with all core files and see if the backtrace is the same? >> >> What is the version of Kamailio? Is it running on a bare metal server or >> a virtual machine/container? >> >> Cheers, >> Daniel >> On 25.02.19 14:21, Abdoul Osséni wrote: >> >> Hello, >> >> Hello dear list, >> >> Today, I have had mutiples crashes. It seems it linked to tm.so module. >> >> -rw------- 1 root kamailio 4299702272 Feb 25 13:08 >> core.kamailio.sig11.29204 >> -rw------- 1 root kamailio 1453023232 Feb 25 13:12 >> core.kamailio.sig11.29203 >> -rw------- 1 root kamailio 1416065024 Feb 25 13:12 >> core.kamailio.sig11.29207 >> -rw------- 1 root kamailio 4299681792 Feb 25 13:16 >> core.kamailio.sig11.19047 >> -rw------- 1 root kamailio 2108506112 Feb 25 13:20 >> core.kamailio.sig11.19043 >> -rw------- 1 root kamailio 4299689984 Feb 25 13:34 >> core.kamailio.sig11.19247 >> -rw------- 1 root kamailio 4299681792 Feb 25 13:34 >> core.kamailio.sig11.19246 >> -rw------- 1 root kamailio 4299698176 Feb 25 13:35 >> core.kamailio.sig11.19248 >> -rw------- 1 root kamailio 4299689984 Feb 25 13:35 >> core.kamailio.sig11.19243 >> -rw------- 1 root kamailio 4299685888 Feb 25 13:35 >> core.kamailio.sig11.19244 >> -rw------- 1 root kamailio 4299689984 Feb 25 13:36 >> core.kamailio.sig11.19242 >> >> root@sbc:/var/cores# gdb /usr/local/sbin/kamailio >> core.kamailio.sig11.29204 >> GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 >> Copyright (C) 2014 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later < >> http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-linux-gnu". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> <http://www.gnu.org/software/gdb/bugs/>. >> Find the GDB manual and other documentation resources online at: >> <http://www.gnu.org/software/gdb/documentation/>. >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from /usr/local/sbin/kamailio...done. >> [New LWP 29204] >> >> warning: .dynamic section for "/lib/x86_64-linux-gnu/libpthread.so.0" is >> not at the expected address (wrong library or version mismatch?) >> Warning: couldn't activate thread debugging using libthread_db: Cannot >> find new threads: generic error >> >> warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading >> has been declined by your `auto-load safe-path' set to >> "$debugdir:$datadir/auto-load". >> To enable execution of this file add >> add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so >> line to your configuration file "/root/.gdbinit". >> To completely disable this security protection add >> set auto-load safe-path / >> line to your configuration file "/root/.gdbinit". >> For more information about this security protection see the >> "Auto-loading safe path" section in the GDB manual. E.g., run from the >> shell: >> info "(gdb)Auto-loading safe path" >> >> warning: Unable to find libthread_db matching inferior's thread library, >> thread debugging will not be available. >> Warning: couldn't activate thread debugging using libthread_db: Cannot >> find new threads: generic error >> >> warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading >> has been declined by your `auto-load safe-path' set to >> "$debugdir:$datadir/auto-load". >> >> warning: Unable to find libthread_db matching inferior's thread library, >> thread debugging will not be available. >> Core was generated by `/usr/local/sbin/kamailio -P >> /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 0x00007f68b9bb4515 in reply_received (p_msg=0x7f693ccd5cc8) at >> t_reply.c:2240 >> 2240 last_uac_status=uac->last_received; >> (gdb) >> >> >> Any ideas? >> >> Regards >> >> Abdoul >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing >> Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >> www.linkedin.com/in/miconda >> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com >> Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in >> Washington, DC, USA -- www.asipto.com >> >> -- > Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- > www.linkedin.com/in/miconda > Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com > Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in > Washington, DC, USA -- www.asipto.com > >
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users