Update after I got access to a system exposing the issue.
The timer process of kamailio was using 100% CPU, being blocked, and by that
not releasing/removing active/terminated transactions, leading to filling the
memory - top output:
```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13582 root 20 0 2274068 18012 12856 R 99.7 0.4 6179:53 kamailio
```
The kamclt ps:
```
{
IDX: 14
PID: 13582
DSC: timer
}
```
Attaching with gdb revealed the following backtrace:
```
#0 0x00007fec0b6ea7d8 in yaSSL::States::What (this=this@entry=0x55b99e2fbf70)
at ./extra/yassl/src/yassl_int.cpp:208
#1 0x00007fec0b6eab31 in yaSSL::SSL::GetError (this=this@entry=0x55b99e2fb600)
at ./extra/yassl/src/yassl_int.cpp:1520
#2 0x00007fec0b6f89fb in yaSSL::receiveData (ssl=..., data=...,
peek=peek@entry=false)
at ./extra/yassl/src/handshake.cpp:1047
#3 0x00007fec0b6de65d in yaSSL::yaSSL_read (ssl=ssl@entry=0x55b99e2fb600,
buffer=buffer@entry=0x55b99e2f91a0,
sz=sz@entry=4) at ./extra/yassl/src/ssl.cpp:363
#4 0x00007fec0b6c6797 in vio_ssl_read (vio=0x55b99e2d83b0, buf=0x55b99e2f91a0
"@", size=<optimized out>)
at ./vio/viossl.c:186
#5 0x00007fec0b69fbae in net_read_raw_loop (count=4, net=0x7fec0c0378f0) at
./sql/net_serv.cc:672
#6 net_read_packet_header (net=0x7fec0c0378f0) at ./sql/net_serv.cc:762
#7 net_read_packet (net=0x7fec0c0378f0, complen=0x7ffe188c6ff0) at
./sql/net_serv.cc:822
#8 0x00007fec0b6a0c2c in my_net_read (net=net@entry=0x7fec0c0378f0) at
./sql/net_serv.cc:899
#9 0x00007fec0b694edc in cli_safe_read_with_ok
(mysql=mysql@entry=0x7fec0c0378f0, parse_ok=parse_ok@entry=0 '\000',
is_data_packet=is_data_packet@entry=0x0) at ./sql-common/client.c:1061
#10 0x00007fec0b69518f in cli_safe_read (mysql=mysql@entry=0x7fec0c0378f0,
is_data_packet=is_data_packet@entry=0x0)
at ./sql-common/client.c:1194
#11 0x00007fec0b695fa8 in cli_read_query_result (mysql=0x7fec0c0378f0) at
./sql-common/client.c:5245
#12 0x00007fec0b697d61 in mysql_real_query (mysql=0x7fec0c0378f0,
query=<optimized out>, length=<optimized out>)
at ./sql-common/client.c:5341
#13 0x00007fec0bc9a2ae in db_mysql_submit_query (_h=0x7fec0c1a4808,
_s=0x7fec0b45a2d0 <sql_str>) at km_dbase.c:116
#14 0x00007fec0b249f6d in db_do_submit_query (_h=0x7fec0c1a4808,
_query=0x7fec0b45a2d0 <sql_str>,
submit_query=0x7fec0bc99b36 <db_mysql_submit_query>) at db_query.c:53
#15 0x00007fec0b24d10d in db_do_delete (_h=0x7fec0c1a4808, _k=0x7ffe188c7310,
_o=0x0, _v=0x7ffe188c7330, _n=2,
val2str=0x7fec0bc98136 <db_mysql_val2str>, submit_query=0x7fec0bc99b36
<db_mysql_submit_query>) at db_query.c:291
#16 0x00007fec0bc9ed64 in db_mysql_delete (_h=0x7fec0c1a4808,
_k=0x7ffe188c7310, _o=0x0, _v=0x7ffe188c7330, _n=2)
at km_dbase.c:515
#17 0x00007fec09669714 in remove_dialog_from_db (cell=0x7feb84c98a98) at
dlg_db_handler.c:714
#18 0x00007fec09645375 in destroy_dlg (dlg=0x7feb84c98a98) at dlg_hash.c:371
#19 0x00007fec0964cca0 in dlg_unref_helper (dlg=0x7feb84c98a98, cnt=2,
fname=0x7fec096dd53f "dlg_handlers.c", fline=410)
at dlg_hash.c:961
#20 0x00007fec09671fb5 in dlg_ontdestroy (t=0x7feb84dbf640, type=131072,
param=0x7ffe188c7620) at dlg_handlers.c:410
#21 0x00007fec099dfb05 in run_trans_callbacks_internal (cb_lst=0x7feb84dbf6b8,
type=131072, trans=0x7feb84dbf640,
params=0x7ffe188c7620) at t_hooks.c:260
#22 0x00007fec099dfc2c in run_trans_callbacks (type=131072,
trans=0x7feb84dbf640, req=0x0, rpl=0x0, code=0)
at t_hooks.c:287
---Type <return> to continue, or q <return> to quit---
#23 0x00007fec09915f45 in free_cell_helper (dead_cell=0x7feb84dbf640, silent=0,
fname=0x7fec09a28882 "timer.c",
fline=689) at h_table.c:166
#24 0x00007fec099cdbf6 in wait_handler (ti=1667374146, wait_tl=0x7feb84dbf6c8,
data=0x7feb84dbf640) at timer.c:689
#25 0x000055b99c9f02c5 in timer_list_expire (t=1667374146, h=0x7feb8461a050,
slow_l=0x7feb8461d988, slow_mark=1904)
at core/timer.c:874
#26 0x000055b99c9f076d in timer_handler () at core/timer.c:939
#27 0x000055b99c9f0c36 in timer_main () at core/timer.c:978
#28 0x000055b99c942322 in main_loop () at main.c:1727
#29 0x000055b99c949a1b in main (argc=7, argv=0x7ffe188c7ca8) at main.c:2696
```
Showing that the processes executing code from inside libmysqlclient, related
to tcp/tls connection. I noticed in config that tls module is used and the
libssl version is 1.1, so this can be a side effect of the issue related to the
new locking API in libssl/libcrypto.
The solutions to try would be either to compile against libssl 1.0.x or use the
pre-loaded library workaround:
*
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared
Run it with one of these two and let's see if it works fine or the issue
appears again.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1895#issuecomment-500895303_______________________________________________
Kamailio (SER) - Development Mailing List
[email protected]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
