Hi Daniel, The file openssl_mutex_shared.so is there and same installation time than the other files!
I have added Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so' To my kamailio.service and I have restarted! I'm sure it will behave better now ;) Tks a lot for the help. Regards, Aymeric Le lun. 16 déc. 2019 à 17:23, Daniel-Constantin Mierla <[email protected]> a écrit : > I pinged Victor to see if he can figure out what happens within the deb > building process that makes the libssl mutex fix not enabled. > > The extra .so preload object should be still installed, try to see if it > is at: > > > /usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so > > Cheers, > Daniel > On 16.12.19 12:09, Aymeric Moizard wrote: > > Good catch! > > As I said in my first mail, I also add the issue with latest 5.2.X so I > suppose the deb package has the same issue for 52X. > > Is the extra binary to load still there? I will check that as soon as I'm > online... > > Tks a lot! > Aymeric > > Le lun. 16 déc. 2019 à 11:16, Daniel-Constantin Mierla <[email protected]> > a écrit : > >> Hello, >> >> for some reason the binary doesn't seem to have the libssl mutex fix, in >> my system with the libssl 1.1 gives: >> >> # kamailio -I >> Print out of kamailio internals >> Version: kamailio 5.3.1 (x86_64/linux) f36ac2 >> Default config: /tmp/kamailio-5.3/etc/kamailio/kamailio.cfg >> Default paths to modules: /tmp/kamailio-5.3/lib64/kamailio/modules >> Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, >> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, >> HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED >> MAX_RECV_BUFFER_SIZE=262144 >> MAX_URI_SIZE=1024 >> BUF_SIZE=65535 >> DEFAULT PKG_SIZE=8MB >> DEFAULT SHM_SIZE=64MB >> ADAPTIVE_WAIT_LOOPS=1024 >> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select >> Source code revision ID: f36ac2 >> Compiled with: gcc 9.2.1 >> Compiled architecture: x86_64 >> Compiled on: 11:11:20 Dec 16 2019 >> Thank you for flying kamailio! >> >> The important part above is the presence of TLS_PTHREAD_MUTEX_SHARED >> compile time flag in the output. >> >> Needs to be investigated why the dep packages have the kamailio binary >> without the libssl mutex fix enabled. >> >> Cheers, >> Daniel >> On 16.12.19 09:22, Aymeric Moizard wrote: >> >> Hi Daniel, >> >> Tks a lot for lookint at it. >> >> $ ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so >> linux-vdso.so.1 (0x00007fff997dd000) >> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 >> (0x00007fe40b53c000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe40b19d000) >> libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 >> (0x00007fe40ad03000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 >> (0x00007fe40aaff000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >> (0x00007fe40a8e2000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fe40ba4a000) >> >> $ /usr/sbin/kamailio -I >> Print out of kamailio internals >> Version: kamailio 5.3.1 (x86_64/linux) >> Default config: /etc/kamailio/kamailio.cfg >> Default paths to modules: /usr/lib/x86_64-linux-gnu/kamailio/modules >> Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, >> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, >> HAVE_RESOLV_RES >> MAX_RECV_BUFFER_SIZE=262144 >> MAX_URI_SIZE=1024 >> BUF_SIZE=65535 >> DEFAULT PKG_SIZE=8MB >> DEFAULT SHM_SIZE=64MB >> ADAPTIVE_WAIT_LOOPS=1024 >> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select >> Source code revision ID: unknown >> Compiled with: gcc 6.3.0 >> Compiled architecture: x86_64 >> Compiled on: >> Thank you for flying kamailio! >> >> Additional note: >> I have tried to better understand the pike module and after reading the >> "end" of the module documentation, >> I do better understand the "Tree of IP" and settings. >> >> The pike documentation, for each settins and description, should refer to >> the section "Chapter 3. Developer Guide", >> otherwise, the parameters cannot be understood. Also, it's not possible >> to understand, according to me, the real time >> for removing an IP from the tree (removing it 100% or only last node of >> IP) >> >> Looking again at my statistics, I feel the first graph is definitly >> showing an issue. This graph is showing >> "$stat(location-users)" and "$stat(location-contacts)". During the 10 >> hours, many users are banned, unregistred, etc.. >> so it is really not expected that the number of registred users is >> maintained. From what I understand, the fact >> that the stats went down when deadlock dissapeared obviouly means >> kamailio threads was in a bad state for the >> last 10 hours... >> >> https://www.antisip.com/sip-antisip-com-register/status2.htm >> >> If you need more information, let me know... >> Regards >> Aymeric >> >> Le lun. 16 déc. 2019 à 08:22, Daniel-Constantin Mierla <[email protected]> >> a écrit : >> >>> Hello, >>> >>> can you provide output of ldd for tls.so and output of "kamailio -I" >>> (that's an uppercase i)? >>> >>> Cheers, >>> Daniel >>> On 13.12.19 16:39, Aymeric Moizard wrote: >>> >>> Hi List, >>> >>> History: >>> * In the past, I had deadlock which was, most probably, related to >>> ssl1.1. >>> We have discussed this issue, and a fix is supposed to workaround the >>> issue that was detected. >>> * With latest 5.2.X, I have experienced ONCE a similar behavior with TCP >>> and TLS being mostly stuck. I have not been using this version much, but >>> the fix was supposed to be in the core of kamailio. >>> >>> The status of the server this night: >>> * I'm today running version: kamailio 5.3.1 (x86_64/linux), >>> * Installed on stretch using http://deb.kamailio.org/kamailio53 >>> repository. >>> * This versions use libssl1.1 >>> * A user reported that he can't connect with TCP >>> * An average of 5000 IPs per 10 minutes are being banned by the pike >>> module >>> (could be twice the same) >>> Yesterday/Today: >>> * at the end of the outage, I had 2479 IP in my ipban htable. (which is >>> equivalent to my statistics showing 2 bans/IP every 10 minutes = 5000) >>> * looking at my logs, it appears that most (ALL?) ip being banned... are >>> my regular users. >>> * looking at my logs, I can't understand why pike would block them. >>> >>> This is a graph for statistics on my service for the last 24 hours: >>> https://www.antisip.com/sip-antisip-com-register/status2.html >>> >>> Yesterday, at 22:18:39, kamailio started to BAN some IPs. 52 IPs were >>> banned in a period of 10 minutes. I can confirm this from my logs. >>> >>> My pike configuration is this one: >>> >>> modparam("pike", "sampling_time_unit", 2) >>> modparam("pike", "reqs_density_per_unit", 64) >>> modparam("pike", "remove_latency", 4) >>> >>> When detecting the issue, this morning, I typed: >>> >>> $> sudo kamctl stats >>> $> sudo kamcmd htable.dump ipban >>> //FAILURE (answer too large...) >>> $> sudo kamctl trap >>> >>> Then, I started an agent with TCP and it worked...??? >>> Then, a few seconds, may be a minute after: >>> >>> $> sudo kamcmd htable.dump ipban >>> //SUCCESS and shows 2479 banned ip. >>> >>> and... everything is back to normal in a few minutes. >>> >>> I haven't restarted kamailio, and all statistics are as expected, as >>> usual. >>> >>> Thus, it looks that " sudo kamctl trap" has triggered something. I >>> already >>> experienced a similar behavior -when testing my ssl1.1 deadlock last >>> year-. >>> >>> 2 questions: >>> 1/ I beleive my "pike" configuration should not ban users. Is my pike >>> configuration wrong? >>> As an example, pike has banned an IP sending one message/second. I >>> believe my configuration should accept that? >>> >>> 2/ Could there still be a TLS issue with libssl1.1? >>> >>> This is the result of the "kamctl trap": >>> >>> https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap >>> >>> Sorry for the long story & hoping to find a long term solution or at >>> least a workaround! >>> >>> Regards >>> Aymeric >>> >>> -- >>> Antisip - http://www.antisip.com >>> >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing >>> [email protected]https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>> >>> -- >>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >>> www.linkedin.com/in/miconda >>> Kamailio World Conference - April 27-29, 2020, in Berlin -- >>> www.kamailioworld.com >>> >>> >> >> -- >> Antisip - http://www.antisip.com >> >> -- >> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >> www.linkedin.com/in/miconda >> Kamailio World Conference - April 27-29, 2020, in Berlin -- >> www.kamailioworld.com >> >> -- > Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- > www.linkedin.com/in/miconda > Kamailio World Conference - April 27-29, 2020, in Berlin -- > www.kamailioworld.com > > -- Antisip - http://www.antisip.com
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
