I tested with docker AlmaLinux 8 and default kamailio.cfg with tls enabled and it does not crash, so indeed the issue can be with snmpops module.
The "previous fragment tail overwritten" is very likely a buffer overflow, which is detected when tls makes some operations with shm, but the previous fragment can be from another module. But having the tail overwritten, makes it hard to jump one fragment back, so the list has to be walked from the beginning. Is it a test system so that you can you provide ssh access to the system? I could try to look at core dump with gdb. Cheers, Daniel On 26.01.23 16:21, Ihor Olkhovskyi wrote: > Daniel, > > Yes, with help of a collegaue was found it's SNMP dealing the > problems. Turning it off helps. > > bt full: > > Core was generated by `/usr/sbin/kamailio -P > /run/kamailio/kamailio.pid -f /etc/kamailio//kamailio.cfg'. > Program terminated with signal SIGABRT, Aborted. > #0 0x00007f5b722feaff in raise () from /lib64/libc.so.6 > Missing separate debuginfos, use: yum debuginfo-install > audit-libs-3.0.7-4.el8.x86_64 bzip2-libs-1.0.6-26.el8.x86_64 > elfutils-libelf-0.187-4.el8.x86_64 glibc-2.28-211.el8.x86_64 > libacl-2.2.53-1.el8.x86_64 libattr-2.4.48-3.el8.x86_64 > libcap-2.48-4.el8.x86_64 libcap-ng-0.7.11-1.el8.x86_64 > libdb-5.3.28-42.el8_4.x86_64 libunistring-0.9.9-3.el8.x86_64 > libxcrypt-4.1.1-6.el8.x86_64 libxml2-2.9.7-15.el8_7.1.x86_64 > libzstd-1.4.4-1.el8.x86_64 > lm_sensors-libs-3.4.0-23.20180522git70f7e08.el8.x86_64 > lua-libs-5.3.4-12.el8.x86_64 mariadb-connector-c-3.1.11-2.el8_3.x86_64 > net-snmp-agent-libs-5.8-25.el8_7.1.x86_64 > net-snmp-libs-5.8-25.el8_7.1.x86_64 openssl-libs-1.1.1k-7.el8_6.x86_64 > pcre-8.42-6.el8.x86_64 perl-libs-5.26.3-421.el8.x86_64 > popt-1.18-1.el8.x86_64 > python2-libs-2.7.18-11.module_el8.7.0+3331+6c5bd851.alma.x86_64 > sqlite-libs-3.26.0-17.el8_7.x86_64 xz-libs-5.2.4-4.el8_6.x86_64 > zlib-1.2.11-21.el8_7.x86_64 > (gdb) bt full > #0 0x00007f5b722feaff in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f5b722d1ea5 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x00000000006eec2f in qm_debug_check_frag (qm=0x7f5b678b4000, > f=0x7f5b678ed820, file=0x7f5b6b926d24 "tls: tls_init.c", line=293, > efile=0x87cc19 "core/mem/q_malloc.c", eline=391) at > core/mem/q_malloc.c:158 > p = 0x7f5b678ed790 > __func__ = "qm_debug_check_frag" > #3 0x00000000006f09a6 in qm_malloc (qmp=0x7f5b678b4000, size=48, > file=0x7f5b6b926d24 "tls: tls_init.c", func=0x7f5b6b928550 > <__func__.23484> "ser_malloc", line=293, mname=0x7f5b6b926d20 "tls") > at core/mem/q_malloc.c:391 > qm = 0x7f5b678b4000 > f = 0x7f5b678ed820 > hash = 11 > list_cntr = 1 > __func__ = "qm_malloc" > #4 0x00000000006fc930 in qm_shm_malloc (qmp=0x7f5b678b4000, size=48, > file=0x7f5b6b926d24 "tls: tls_init.c", func=0x7f5b6b928550 > <__func__.23484> "ser_malloc", line=293, mname=0x7f5b6b926d20 "tls") > at core/mem/q_malloc.c:1301 > r = 0x7f5b667d1464 <sc_find_auth_alg_byoid+116> > #5 0x00007f5b6b8d3c31 in ser_malloc (size=48, fname=0x7f5b70676bcb > "crypto/evp/digest.c", fline=54) at tls_init.c:293 > __func__ = "ser_malloc" > #6 0x00007f5b705b890d in CRYPTO_zalloc () from /lib64/libcrypto.so.1.1 > No symbol table info available. > #7 0x00007f5b667d2163 in sc_hash_type () from /lib64/libnetsnmp.so.35 > No symbol table info available. > #8 0x00007f5b667ce9df in hash_engineID () from /lib64/libnetsnmp.so.35 > No symbol table info available. > #9 0x00007f5b667ceb11 in search_enginetime_list () from > /lib64/libnetsnmp.so.35 > No symbol table info available. > #10 0x00007f5b667cef8e in set_enginetime () from /lib64/libnetsnmp.so.35 > No symbol table info available. > #11 0x00007f5b667ce885 in init_snmpv3_post_config () from > /lib64/libnetsnmp.so.35 > No symbol table info available. > #12 0x00007f5b667d3c92 in snmp_call_callbacks () from > /lib64/libnetsnmp.so.35 > No symbol table info available. > #13 0x00007f5b6725001b in register_with_master_agent > (name_to_register_under=0x7f5b6726dbad "snmpstats_alarm_agent") at > sub_agent.c:196 > __func__ = "register_with_master_agent" > #14 0x00007f5b67251f14 in run_alarm_check (ticks=111051753, attr=0x0) > at alarm_checks.c:102 > msg_queue_minor_threshold = 0 > msg_queue_major_threshold = 0 > dialog_minor_threshold = 0 > dialog_major_threshold = 0 > firstRun = 1 '\001' > bytesInMsgQueue = 0 > numActiveDialogs = 8820307 > #15 0x0000000000531cde in compat_old_handler (ti=1776828050, > tl=0x7f5b67e00120, data=0x7f5b67e00120) at core/timer.c:980 > t = 0x7f5b67e00120 > #16 0x000000000053284e in slow_timer_main () at core/timer.c:1105 > n = 12 > ret = 4294967295 > tl = 0x7f5b67e00120 > i = 4 > __func__ = "slow_timer_main" > #17 0x000000000042e3e1 in main_loop () at main.c:1815 > i = 4 > pid = 0 > si = 0x0 > si_desc = "udp receiver child=3 sock=188.185.121.232:5060 > <http://188.185.121.232:5060> (dev.tone.cern.ch:5060 > <http://dev.tone.cern.ch:5060>)\000\000(\274tq[\177\000\000S\226\206\000\000\000\000\000\020\064\227q[\177\000\000\000\000\000\004\000\000\000\000\060\247=r[\177\000\000\060\000\000\000\060\000\000\000H\365$)\377\177\000" > nrprocs = 4 > woneinit = 1 > __func__ = "main_loop" > #18 0x0000000000438d14 in main (argc=5, argv=0x7fff2924fa68) at > main.c:3078 > cfg_stream = 0x24002a0 > c = -1 > r = 0 > tmp = 0x7fff2924f8c0 "\377\377\377\377" > tmp_len = 0 > port = 0 > proto = 32603 > ahost = 0x0 > aport = 0 > options = 0x8114d8 > ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" > ret = -1 > seed = 1009486776 > --Type <RET> for more, q to quit, c to continue without paging-- > rfd = 4 > debug_save = 0 > debug_flag = 0 > dont_fork_cnt = 0 > n_lst = 0x7fff2924fa98 > p = 0x7f5b7241a838 <_dl_vdso_vsym+104> > "H\213t$\030ZYH\205\366t\026f\203~\006\361t-H\205\300t(H\213" > st = {st_dev = 24, st_ino = 84031, st_nlink = 2, st_mode = > 16832, st_uid = 994, st_gid = 991, __pad0 = 0, st_rdev = 0, st_size = > 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1674662431, > tv_nsec = 788813597}, st_mtim = { > tv_sec = 1674746240, tv_nsec = 961870546}, st_ctim = > {tv_sec = 1674746240, tv_nsec = 961870546}, __glibc_reserved = {0, 0, 0}} > tbuf = > "(O\003s[\177\000\000\000\000\000\000\000\000\000\000\370\210%s[\177\000\000(O\003s[\177\000\000\200\366$)\377\177\000\000`G\003s[\177\000\000x5&s[\177\000\000\000@\003s[\177\000\000\274I\003s[\177\000\000\263/\004s[\177\000\000\350/+r[\177\000\000\254\202\226\006\000\000\000\000\036\215\005s[\177\000\000\254\202\226\006\000\000\000\000\205\317c\t\000\000\000\000y0\004s[\177\000\000\020\215\005s[\177\000\000\354\032+r[\177\000\000\000\000\000\000\000\000\000\000\220\a&s[\177\000\000\001\000\000\000\000\000\000\000\002\035\326D\224\266\000\000\020\062&s[\177\000\000\200\224%s[\177\000\000\020\062&s[\177\000\000"... > option_index = 0 > long_options = {{name = 0x813876 "help", has_arg = 0, flag = > 0x0, val = 104}, {name = 0x80e6f4 "version", has_arg = 0, flag = 0x0, > val = 118}, {name = 0x81387b "alias", has_arg = 1, flag = 0x0, val = > 1024}, {name = 0x813881 "subst", > has_arg = 1, flag = 0x0, val = 1025}, {name = 0x813887 > "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x813890 > "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x81389a > "server-id", has_arg = 1, flag = 0x0, > val = 1028}, {name = 0x8138a4 "loadmodule", has_arg = 1, > flag = 0x0, val = 1029}, {name = 0x8138af "modparam", has_arg = 1, > flag = 0x0, val = 1030}, {name = 0x8138b8 "log-engine", has_arg = 1, > flag = 0x0, val = 1031}, { > name = 0x8138c3 "debug", has_arg = 1, flag = 0x0, val = > 1032}, {name = 0x8138c9 "cfg-print", has_arg = 0, flag = 0x0, val = > 1033}, {name = 0x8138d3 "atexit", has_arg = 1, flag = 0x0, val = > 1034}, {name = 0x0, has_arg = 0, flag = 0x0, > val = 0}} > __func__ = "main" > > If it helps. > Many thanks! > > Le jeu. 26 janv. 2023 à 16:06, Daniel-Constantin Mierla > <[email protected]> a écrit : > > Hello, > > it seems to be at start up, does it happen every time? > > Cheers, > Daniel > > On 26.01.23 14:43, Ihor Olkhovskyi wrote: >> Hello! >> >> Currently testing Kamailio on Almalinux 8 (stated to be RHEL 8 >> binary compatible) and got this while trying to get TLS working: >> >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:347]: >> ksr_tls_fill_missing(): TLSc<default>: ca_list='(null)' >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:354]: >> ksr_tls_fill_missing(): TLSc<default>: ca_path='(null)' >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:361]: >> ksr_tls_fill_missing(): TLSc<default>: crl='(null)' >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:365]: >> ksr_tls_fill_missing(): TLSc<default>: require_certificate=1 >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:372]: >> ksr_tls_fill_missing(): TLSc<default>: cipher_list='(null)' >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:379]: >> ksr_tls_fill_missing(): TLSc<default>: private_key='(null)' >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:383]: >> ksr_tls_fill_missing(): TLSc<default>: verify_certificate=1 >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:386]: >> ksr_tls_fill_missing(): TLSc<default>: verify_depth=9 >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:389]: >> ksr_tls_fill_missing(): TLSc<default>: verify_client=0 >> /usr/sbin/kamailio[32858]: INFO: tls [tls_domain.c:736]: >> set_verification(): TLSc<default>: Server MUST present valid >> certificate >> /usr/sbin/kamailio[32862]: ERROR: rtpengine [rtpengine.c:3070]: >> send_rtpp_command(): can't send command "ping" to RTPEngine >> <udp:127.0.0.1:19999 <http://127.0.0.1:19999>> >> /usr/sbin/kamailio[32862]: ERROR: rtpengine [rtpengine.c:2919]: >> rtpp_test(): proxy did not respond to ping >> /usr/sbin/kamailio[32883]: INFO: snmpstats [snmpstats.c:519]: >> spawn_sysUpTime_child(): A snmpVersion parameter was not >> provided. Defaulting to 3 >> /usr/sbin/kamailio[32883]: ERROR: snmpstats [snmpstats.c:561]: >> spawn_sysUpTime_child(): snmpget failed to run. Did you supply >> the snmpstats module with a proper snmpgetPath parameter? The >> kamailioSIPServiceStartTime is defaulting to zero >> /usr/sbin/kamailio[32881]: INFO: ctl [io_listener.c:214]: >> io_listen_loop(): io_listen_loop: using epoll_lt io watch method >> (config) >> /usr/sbin/kamailio[32870]: CRITICAL: <core> >> [core/mem/q_malloc.c:145]: qm_debug_check_frag(): BUG: qm: prev. >> fragm. tail overwritten(0, >> abcdefed)[0x7fe8375d3820:0x7fe8375d3858]! Memory allocator was >> called from tls: tls_init.c:293. Fragment marked by tls: >> tls_init.c:293. Exec from core/mem/q_malloc.c:391. >> /usr/sbin/kamailio[32870]: CRITICAL: <core> >> [core/mem/q_malloc.c:153]: qm_debug_check_frag(): BUG: qm: prev. >> fragm. tail overwritten [0x7fe8375d3790:0x7fe8375d37c8] - >> fragment marked by tls: tls_init.c:293 >> systemd[1]: Started Process Core Dump (PID 32923/UID 0). >> systemd-coredump[32924]: Process 32870 (kamailio) of user 0 >> dumped core.#012#012Stack trace of thread 32870:#012#0 >> 0x00007fe841786aff raise (libc.so.6)#012#1 0x00007fe841759ea5 >> abort (libc.so.6)#012#2 0x00000000006eec2f qm_debug_check_frag >> (kamailio)#012#3 0x00000000006f09a6 qm_malloc (kamailio)#012#4 >> 0x00000000006fc930 qm_shm_malloc (kamailio)#012#5 >> 0x00007fe83b5b9c31 ser_malloc (tls.so)#012#6 0x00007fe83fa4090d >> CRYPTO_zalloc (libcrypto.so.1.1)#012#7 0x00007fe8364b8163 >> sc_hash_type (libnetsnmp.so.35)#012#8 0x00007fe8364b49df >> hash_engineID (libnetsnmp.so.35)#012#9 0x00007fe8364b4b11 >> search_enginetime_list (libnetsnmp.so.35)#012#10 >> 0x00007fe8364b4f8e set_enginetime (libnetsnmp.so.35)#012#11 >> 0x00007fe8364b4885 init_snmpv3_post_config >> (libnetsnmp.so.35)#012#12 0x00007fe8364b9c92 snmp_call_callbacks >> (libnetsnmp.so.35)#012#13 0x00007fe836f3601b >> register_with_master_agent (snmpstats.so)#012#14 >> 0x00007fe836f37f14 run_alarm_check (snmpstats.so)#012#15 >> 0x0000000000531cde compat_old_handler (kamailio)#012#16 >> 0x000000000053284e slow_timer_main (kamailio)#012#17 >> 0x000000000042e3e1 main_loop (kamailio)#012#18 0x0000000000438d14 >> main (kamailio)#012#19 0x00007fe841772d85 __libc_start_main >> (libc.so.6)#012#20 0x000000000041cfce _start (kamailio) >> /usr/sbin/kamailio[32915]: CRITICAL: <core> [core/pass_fd.c:277]: >> receive_fd(): EOF on 24 >> /usr/sbin/kamailio[32858]: ALERT: <core> [main.c:774]: >> handle_sigs(): child process 32870 exited by a signal 6 >> >> ... >> version: kamailio 5.6.3 (x86_64/linux) fe0c4d >> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, >> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, >> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, >> USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, >> TLS_PTHREAD_MUTEX_SHARED >> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, >> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >> id: fe0c4d >> compiled on 11:02:26 Jan 19 2023 with gcc 8.5.0 >> >> Same config is starting without issues on CentOS 7. >> >> OpenSSL version is 1.1.1k >> Kamailio is installed from official RPM repos. >> >> Is core dump would be useful here or best to compile from source >> and redo? >> >> -- >> Best regards, >> Ihor (Igor) >> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> To unsubscribe send an email to [email protected] >> Important: keep the mailing list in the recipients, do not reply only to >> the sender! >> Edit mailing list options or unsubscribe: > > -- > Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> -- > www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Kamailio World Conference - June 5-7, 2023 - www.kamailioworld.com > <http://www.kamailioworld.com> > > > > -- > Best regards, > Ihor (Igor) > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to [email protected] > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - June 5-7, 2023 - www.kamailioworld.com
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
