You can download the code from git repository and build the openssl_mutex_shared.so locally.
Or install from the nightly builts, there should be the version with the fix embedded -- after installation check kamailio -I and see if it lists TLS_PTHREAD_MUTEX_SHARED. Cheers, Daniel On 06.01.20 16:48, Andrew Chen wrote: > I have seen similar issues with pike and running 5.2.5 installed using > deb repo. I don't see this openssl_mutex_shared directory: > > root@ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio # ls -lart > total 400 > -rw-r--r-- 1 root root 22528 Oct 10 12:29 libtrie.so.1.0 > lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so.1 -> libtrie.so.1.0 > lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so -> libtrie.so.1.0 > -rw-r--r-- 1 root root 63864 Oct 10 12:29 libsrutils.so.1.0 > lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so.1 -> > libsrutils.so.1.0 > lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so -> > libsrutils.so.1.0 > -rw-r--r-- 1 root root 51472 Oct 10 12:29 libsrdb2.so.1.0 > lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so.1 -> > libsrdb2.so.1.0 > lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so -> libsrdb2.so.1.0 > -rw-r--r-- 1 root root 211264 Oct 10 12:29 libsrdb1.so.1.0 > lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so.1 -> > libsrdb1.so.1.0 > lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so -> libsrdb1.so.1.0 > drwxr-xr-x 4 root root 4096 Dec 3 21:32 . > drwxr-xr-x 3 root root 4096 Dec 3 21:33 kamctl > drwxr-xr-x 2 root root 4096 Dec 3 21:33 modules > drwxr-xr-x 47 root root 36864 Dec 10 20:13 .. > root@ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio # > > and here are my command outputs: > > root@ashmainkama51:~ # ldd > /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so > linux-vdso.so.1 (0x00007ffd28de0000) > libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 > (0x00007f925de6d000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f925da7c000) > libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 > (0x00007f925d5b1000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f925d392000) > /lib64/ld-linux-x86-64.so.2 (0x00007f925e39d000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007f925d18e000) > root@ashmainkama51:~ # kamailio -I > Print out of kamailio internals > Version: kamailio 5.2.5 (x86_64/linux) > Default config: /etc/kamailio/kamailio.cfg > Default paths to modules: /usr/lib/x86_64-linux-gnu/kamailio/modules > Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, > USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, > SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, > USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, > USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES > MAX_RECV_BUFFER_SIZE=262144 > MAX_URI_SIZE=1024 > BUF_SIZE=65535 > DEFAULT PKG_SIZE=8MB > DEFAULT SHM_SIZE=64MB > ADAPTIVE_WAIT_LOOPS=1024 > TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select > Source code revision ID: unknown > Compiled with: gcc 7.4.0 > Compiled on: > Thank you for flying kamailio! > > > On Mon, Dec 16, 2019 at 12:51 PM Aymeric Moizard <[email protected] > <mailto:[email protected]>> wrote: > > Hi Daniel, > > The file openssl_mutex_shared.so is there and same installation > time than the other files! > > I have added > > > Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so' > > To my kamailio.service and I have restarted! > I'm sure it will behave better now ;) > > Tks a lot for the help. > Regards, > Aymeric > > > Le lun. 16 déc. 2019 à 17:23, Daniel-Constantin Mierla > <[email protected] <mailto:[email protected]>> a écrit : > > I pinged Victor to see if he can figure out what happens > within the deb building process that makes the libssl mutex > fix not enabled. > > The extra .so preload object should be still installed, try to > see if it is at: > > > /usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so > > Cheers, > Daniel > > On 16.12.19 12:09, Aymeric Moizard wrote: >> Good catch! >> >> As I said in my first mail, I also add the issue with latest >> 5.2.X so I suppose the deb package has the same issue for 52X. >> >> Is the extra binary to load still there? I will check that as >> soon as I'm online... >> >> Tks a lot! >> Aymeric >> >> Le lun. 16 déc. 2019 à 11:16, Daniel-Constantin Mierla >> <[email protected] <mailto:[email protected]>> a écrit : >> >> Hello, >> >> for some reason the binary doesn't seem to have the >> libssl mutex fix, in my system with the libssl 1.1 gives: >> >> # kamailio -I >> Print out of kamailio internals >> Version: kamailio 5.3.1 (x86_64/linux) f36ac2 >> Default config: /tmp/kamailio-5.3/etc/kamailio/kamailio.cfg >> Default paths to modules: >> /tmp/kamailio-5.3/lib64/kamailio/modules >> Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, >> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, >> SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, >> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >> USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED >> MAX_RECV_BUFFER_SIZE=262144 >> MAX_URI_SIZE=1024 >> BUF_SIZE=65535 >> DEFAULT PKG_SIZE=8MB >> DEFAULT SHM_SIZE=64MB >> ADAPTIVE_WAIT_LOOPS=1024 >> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, >> select >> Source code revision ID: f36ac2 >> Compiled with: gcc 9.2.1 >> Compiled architecture: x86_64 >> Compiled on: 11:11:20 Dec 16 2019 >> Thank you for flying kamailio! >> >> The important part above is the presence of >> TLS_PTHREAD_MUTEX_SHARED compile time flag in the output. >> >> Needs to be investigated why the dep packages have the >> kamailio binary without the libssl mutex fix enabled. >> >> Cheers, >> Daniel >> >> On 16.12.19 09:22, Aymeric Moizard wrote: >>> Hi Daniel, >>> >>> Tks a lot for lookint at it. >>> >>> $ ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so >>> linux-vdso.so.1 (0x00007fff997dd000) >>> libssl.so.1.1 => >>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fe40b53c000) >>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 >>> (0x00007fe40b19d000) >>> libcrypto.so.1.1 => >>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 >>> (0x00007fe40ad03000) >>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >>> (0x00007fe40aaff000) >>> libpthread.so.0 => >>> /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe40a8e2000) >>> /lib64/ld-linux-x86-64.so.2 (0x00007fe40ba4a000) >>> >>> $ /usr/sbin/kamailio -I >>> Print out of kamailio internals >>> Version: kamailio 5.3.1 (x86_64/linux) >>> Default config: /etc/kamailio/kamailio.cfg >>> Default paths to modules: >>> /usr/lib/x86_64-linux-gnu/kamailio/modules >>> Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, >>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, >>> SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, >>> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >>> USE_DST_BLACKLIST, HAVE_RESOLV_RES >>> MAX_RECV_BUFFER_SIZE=262144 >>> MAX_URI_SIZE=1024 >>> BUF_SIZE=65535 >>> DEFAULT PKG_SIZE=8MB >>> DEFAULT SHM_SIZE=64MB >>> ADAPTIVE_WAIT_LOOPS=1024 >>> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, >>> select >>> Source code revision ID: unknown >>> Compiled with: gcc 6.3.0 >>> Compiled architecture: x86_64 >>> Compiled on: >>> Thank you for flying kamailio! >>> >>> Additional note: >>> I have tried to better understand the pike module and >>> after reading the "end" of the module documentation, >>> I do better understand the "Tree of IP" and settings. >>> >>> The pike documentation, for each settins and >>> description, should refer to the section "Chapter 3. >>> Developer Guide", >>> otherwise, the parameters cannot be understood. Also, >>> it's not possible to understand, according to me, the >>> real time >>> for removing an IP from the tree (removing it 100% or >>> only last node of IP) >>> >>> Looking again at my statistics, I feel the first graph >>> is definitly showing an issue. This graph is showing >>> "$stat(location-users)" and "$stat(location-contacts)". >>> During the 10 hours, many users are banned, unregistred, >>> etc.. >>> so it is really not expected that the number of >>> registred users is maintained. From what I understand, >>> the fact >>> that the stats went down when deadlock dissapeared >>> obviouly means kamailio threads was in a bad state for the >>> last 10 hours... >>> >>> https://www.antisip.com/sip-antisip-com-register/status2.htm >>> >>> If you need more information, let me know... >>> Regards >>> Aymeric >>> >>> Le lun. 16 déc. 2019 à 08:22, Daniel-Constantin Mierla >>> <[email protected] <mailto:[email protected]>> a écrit : >>> >>> Hello, >>> >>> can you provide output of ldd for tls.so and output >>> of "kamailio -I" (that's an uppercase i)? >>> >>> Cheers, >>> Daniel >>> >>> On 13.12.19 16:39, Aymeric Moizard wrote: >>>> Hi List, >>>> >>>> History: >>>> * In the past, I had deadlock which was, most >>>> probably, related to ssl1.1. >>>> We have discussed this issue, and a fix is >>>> supposed to workaround the issue that was detected. >>>> * With latest 5.2.X, I have experienced ONCE a >>>> similar behavior with TCP and TLS being mostly >>>> stuck. I have not been using this version much, but >>>> the fix was supposed to be in the core of kamailio. >>>> >>>> The status of the server this night: >>>> * I'm today running version: kamailio 5.3.1 >>>> (x86_64/linux), >>>> * Installed on stretch using >>>> http://deb.kamailio.org/kamailio53 repository. >>>> * This versions use libssl1.1 >>>> * A user reported that he can't connect with TCP >>>> * An average of 5000 IPs per 10 minutes are being >>>> banned by the pike module >>>> (could be twice the same) >>>> Yesterday/Today: >>>> * at the end of the outage, I had 2479 IP in my >>>> ipban htable. (which is equivalent to my statistics >>>> showing 2 bans/IP every 10 minutes = 5000) >>>> * looking at my logs, it appears that most (ALL?) >>>> ip being banned... are my regular users. >>>> * looking at my logs, I can't understand why pike >>>> would block them. >>>> >>>> This is a graph for statistics on my service for >>>> the last 24 hours: >>>> >>>> https://www.antisip.com/sip-antisip-com-register/status2.html >>>> >>>> Yesterday, at 22:18:39, kamailio started to BAN >>>> some IPs. 52 IPs were banned in a period of 10 >>>> minutes. I can confirm this from my logs. >>>> >>>> My pike configuration is this one: >>>> >>>> modparam("pike", "sampling_time_unit", 2) >>>> modparam("pike", "reqs_density_per_unit", 64) >>>> modparam("pike", "remove_latency", 4) >>>> >>>> When detecting the issue, this morning, I typed: >>>> >>>> $> sudo kamctl stats >>>> $> sudo kamcmd htable.dump ipban >>>> //FAILURE (answer too large...) >>>> $> sudo kamctl trap >>>> >>>> Then, I started an agent with TCP and it worked...??? >>>> Then, a few seconds, may be a minute after: >>>> >>>> $> sudo kamcmd htable.dump ipban >>>> //SUCCESS and shows 2479 banned ip. >>>> >>>> and... everything is back to normal in a few minutes. >>>> >>>> I haven't restarted kamailio, and all statistics >>>> are as expected, as usual. >>>> >>>> Thus, it looks that " sudo kamctl trap" has >>>> triggered something. I already >>>> experienced a similar behavior -when testing my >>>> ssl1.1 deadlock last year-. >>>> >>>> 2 questions: >>>> 1/ I beleive my "pike" configuration should not ban >>>> users. Is my pike configuration wrong? >>>> As an example, pike has banned an IP sending one >>>> message/second. I believe my configuration should >>>> accept that? >>>> >>>> 2/ Could there still be a TLS issue with libssl1.1? >>>> >>>> This is the result of the "kamctl trap": >>>> >>>> >>>> https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap >>>> >>>> Sorry for the long story & hoping to find a long >>>> term solution or at least a workaround! >>>> >>>> Regards >>>> Aymeric >>>> >>>> -- >>>> Antisip - http://www.antisip.com >>>> >>>> _______________________________________________ >>>> Kamailio (SER) - Users Mailing List >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>> >>> -- >>> Daniel-Constantin Mierla -- www.asipto.com >>> <http://www.asipto.com> >>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >>> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >>> Kamailio World Conference - April 27-29, 2020, in Berlin -- >>> www.kamailioworld.com <http://www.kamailioworld.com> >>> >>> >>> >>> -- >>> Antisip - http://www.antisip.com >> >> -- >> Daniel-Constantin Mierla -- www.asipto.com >> <http://www.asipto.com> >> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >> Kamailio World Conference - April 27-29, 2020, in Berlin -- >> www.kamailioworld.com <http://www.kamailioworld.com> >> > -- > Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> -- > www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Kamailio World Conference - April 27-29, 2020, in Berlin -- > www.kamailioworld.com <http://www.kamailioworld.com> > > > > -- > Antisip - http://www.antisip.com > _______________________________________________ > Kamailio (SER) - Users Mailing List > [email protected] <mailto:[email protected]> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > > -- > Andy Chen > Sr. Telephony Lead Engineer > 415 516 5535 (M) > achen@ <mailto:[email protected]>fuze.com <http://fuze.com> > > > *Confidentiality Notice: The information contained in this e-mail and any > attachments may be confidential. If you are not an intended recipient, you > are hereby notified that any dissemination, distribution or copying of > this > e-mail is strictly prohibited. If you have received this e-mail in error, > please notify the sender and permanently delete the e-mail and any > attachments immediately. You should not retain, copy or use this e-mail or > any attachment for any purpose, nor disclose all or any part of the > contents to any other person. Thank you.* > > _______________________________________________ > Kamailio (SER) - Users Mailing List > [email protected] > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
