Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

On Debian stable I'm periodically getting following error: ``` /usr/sbin/kamailio[13996]: ERROR: tls [tls_server.c:192]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7494944 threshold1=7864320 ``` -- You are receiving this because you were

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

This is likely related to libmysqlclient changes for ubuntu, like reported by #2394. On Debian stable I was not able to reproduce on a couple of heavy duty deployments, so might be something different. I am closing this one, if you get it on Debian stable again, open a new issue and attach the

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Closed #1895. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#event-3642402457___ Kamailio (SER) - Development Mailing List

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

It seems that kamailio in Debian stable (5.2.1-1) is affected by this bug too. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Are there any other versions affected anyone knows of ? On Fri, Nov 15, 2019 at 5:10 AM Jay J wrote: > @henningw we have stuck to db_mode 0, so > the issue no longer resurfaces. It is still present, however, with db_mode > 1. > > — > You are receiving this because

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@henningw we have stuck to db_mode 0, so the issue no longer resurfaces. It is still present, however, with db_mode 1. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@henningw we have stuck to db_mode 0, so the issue no longer resurfaces. It is still present, however, with db_mode 1. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@Jaykah - any update on this one? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-554320631___ Kamailio (SER) - Development Mailing

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

As I looked again over the issue, I see in the backtrace (I attached) that the mysql client library is using yassl not libssl (which I expected). @Jaykah - do you know if the library is installed from the default ubuntu repository or is something customized for RDS? Also, for you backtrace,

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Hi @henningw - yes, I do. I saw a similar problem on the mailing list and we are also using RDS - perhaps that's the issue (deadlock?). It looks like this happens on a record delete for the dialog - looks like db_mode 0 resolves this - so far instances with that option enabled have not

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@Jaykah - do you still see this error, maybe after updating to a new kamailio version? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

(gdb) bt n #0 0x7fab34880860 in __errno_location@plt () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #1 0x7fab348bb589 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #2 0x7fab348bb849 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #3

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

strings /usr/lib/x86_64-linux-gnu/libcrypto.so | grep "^OpenSSL" OpenSSLDie OpenSSL_add_all_ciphers OpenSSL_add_all_digests OpenSSL 1.0.2n 7 Dec 2017 OpenSSL HMAC method OpenSSL EC algorithm OpenSSL RSA method OpenSSL DSA method OpenSSL ECDSA method OpenSSL DH Method OpenSSL X9.42 DH method

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

FYI- we have to build our own debian9 pkgs due to the openssl mem leak issues not sure why this isn't in the kamailio's debian build system yet because im pretty sure there is an issue on GH related to it this is from our `build.sh` container: # fix for libssl bug apt-get -y purge libssl-dev &&

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Adding also that the mysql timeout might not work always, as presented in manual: * https://dev.mysql.com/doc/refman/5.5/en/mysql-options.html Citing: ``` MYSQL_OPT_READ_TIMEOUT (argument type: unsigned int *) The timeout in seconds for each attempt to read from the server. There are

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Update after I got access to a system exposing the issue. The timer process of kamailio was using 100% CPU, being blocked, and by that not releasing/removing active/terminated transactions, leading to filling the memory - top output: ``` PID USER PR NIVIRTRESSHR S %CPU

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

You can write to miconda [at] gmail.com -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-493864863___ Kamailio (SER) - Development

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda yes, I do have a machine where the issue is reproduced, where can I email the information? If you could reply to that email with your public key I'll add that right away as well. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Is there a way to get access to the system exposing the issue and do some troubleshooting there (it may require sudo privileges)? The leak is identified by the number of allocated memory chunks by uac, with side effects to transactions, but the reason for it is yet unknown. With Kamailio World

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda just wanted to follow up on this - dumping those records should show us what the issue might be, how can that be inspected? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda what would be the best way to dump the kamailio uac memory segments to see those records and compare with actual registrations? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@henningw will do that instead, thanks! -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-487394400___ Kamailio (SER) - Development

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@Jaykah - bug bounties are not really common in the Kamailio project so far. Commercial support options can be found on e.g. in the directory on kamailio.org. I will write you also an email with some more information. -- You are receiving this because you were mentioned. Reply to this email

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Upping bounty to $250 -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-487283380___ Kamailio (SER) - Development Mailing List

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda We got rid of the trunks/accounts causing 500 errors and auth failure, yet are still seeing the leak in the kamailio process. Any other way to troubleshoot this? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda ~2000 matches for "authentication failed for" ~2800 matches for "got sip response 500 while registering" -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@Jaykah - if they repeat, is important to see how many times -- can you count how many error messages that contain the next texts are from start to out of memory logs? ``` authentication failed for got sip response 500 while registering ``` Something like: ``` grep "authentication failed for"

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@kamailio-sync could you please clarify? Not sure I understand -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-484844338___ Kamailio

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Have you tried moving to db and see what happens? On Thu, 18 Apr 2019 at 18:26, Jay J wrote: > also doubling the bounty for this, hoping the community would help us get > a stable 5.1.7+ > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly,

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

also doubling the bounty for this, hoping the community would help us get a stable 5.1.7+ -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda what I meant is the non-expiring registration records stored in memory As to the error messages, those are about it, everything else is repetitive and the remaining logs consist of registrations, test calls, etc. Perhaps there's a good way to export all shm data related to uac? --

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda the number of records in uacreg is significantly less than that, that makes me wonder if the hash table cleanup is to blame ``` Apr 16 00:05:37 ip-10-0-0-0 kamailio[12057]: ERROR: uac [uac_reg.c:913]: uac_reg_tm_callback(): authentication failed for Apr 16 00:05:37 ip-10-0-0-0

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Allocated size for tm is indeed very high and it also looks like being induced from uac module, because the uac_reg_update() allocates size of UUID (less than 100) for each registration update, overall being 11472688, means over 100k remote registrations. Do you have so many records in you

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

With busy traveling during March, this issue got out of my sight. It looks like a problem with TM UAC, the allocated size there is big. Could be related to uac_reg (uac remote registrations), I will dig into it a bit. -- You are receiving this because you are subscribed to this thread. Reply

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Posting a $100 bounty on this issue if anyone wants to take a look. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

We also have a test system with the issue reproduced if a closer look is needed @miconda -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda any other info I can gather that would help locate the issue here? The issue is recurring. Thank you -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

Another occurence: ``` 11(8342) ERROR: usrloc [ucontact.c:100]: new_ucontact(): no more shm memory 11(8342) ERROR: usrloc [urecord.c:149]: mem_insert_ucontact(): failed to create new contact 11(8342) ERROR: usrloc [urecord.c:573]: insert_ucontact(): failed to insert contact 11(8342) ERROR:

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

@miconda -m 2048 -M 16 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1895#issuecomment-474684038___ Kamailio (SER) - Development

Re: [sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

It is the shared memory that runs out of space. As I can see it, you provided the stats for pkg. You have to take the ones for shm. What are the values you give for `-m` and `-M` command line parameters for kamailio? -- You are receiving this because you are subscribed to this thread. Reply

[sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.4 and 5.1.7 (#1895)

### Description Kamailio runs out of memory and stops processing requests. ``` ERROR: [core/tcp_main.c:973]: tcpconn_new(): mem. allocation failure ERROR: [core/tcp_main.c:4042]: handle_new_connect(): tcpconn_new failed, closing socket ERROR: uac [uac_reg.c:533]: reg_ht_add(): no more shm