Hi!

I haven't used the workaround yet: I'm focusing on trying to make sure I
have the same issue
or trying to figure out how to force it to happen.

I have started to check again the server today and I started by this
command:
 $> sudo kamcmd tls.list

In my previous description, the above was a dead lock. Today, It finally
completed, but
after 5 minutes. (I suspect 5 minutes is abnormal)

During the long running command:
-> UDP was working
-> TCP was not:
-> The TCP connection is being ESTABLISHED, but the SIP message was not
replied.
    (this was the behavior I had before)

At the same time, I took a trap "sudo kamctl trap". (during the dead lock)
-> one thread is on "tls_list" (tls_rpc.c:154)
-> one thread is on tcpconn_get (core/tcp_main.c:1449) called from tcp_send
(core/tcp_main.c:1716)
    and seems to be sending a 484 Address Incomplete on a TLS connection
-> 2 threads are on CRYPTO_THREAD_write_lock on a backtrace showing
"SSL_do_handshake/tls_accept"

Suddenly, "sudo kamcmd tls.list" completed, and then, my TCP Agent received
4 answers from kamailio for the last 4 REGISTER sent.

I have a network capture for my TCP agent.
I have a trap showing 2 thread waiting on "CRYPTO_THREAD_write_lock"

Conclusion:
The use-case showed that the lock was VERY long.
The use-case showed that the lock was TEMPORARY...

Side-note: From my understanding of the multi-fork/openssl issue, I would
expect
to see dead lock happening very fast after a kamailio restart?

Do you expect the preload workaround to work in such behavior?
Or do you consider that my issue is different?

Because there is no "real" dead-lock, I don't understand why "my" issue
would be related to libssl1.1...

My gdb trap, network capture are available in private exchange if you need!
(please ask me by direct email)

Tks
Aymeric

Le lun. 13 mai 2019 à 12:48, Daniel-Constantin Mierla <mico...@gmail.com> a
écrit :

> Hello,
>
> thanks for the feedback! It is good to know that it works well so far for
> you. I don't see any reason not to make the library to preload as part of
> the next release.
>
> Just to let everyone know, for now, the built packages are pinned to link
> against libssl 1.0.x.
>
> Soon, I will approach the openssl project in order to find a proper
> solution for long term.
>
> Cheers,
> Daniel
> On 13.05.19 10:48, Floimair Florian wrote:
>
> Hi all!
>
>
>
> We have used the work-around with the pre-loaded library and so far this
> seems to have fixed our problem (that my colleague Kristijan Vrban
> reported).
>
> At least we did not have a single failure within the last week, whereas
> before the issue happened about once every 2 days.
>
> Would be nice if this would be part of the next Kamailio version.
>
>
>
>
>
>
>
> With best regards
>
>
>
> *Florian Floimair *Innovation - Software-Development
>
>
> *COMMEND INTERNATIONAL GMBH *A-5020 Salzburg, Saalachstraße 51
> http://www.commend.com
>
>
>
> *Security and Communication by Commend *FN 178618z | LG Salzburg
>
>
>
> *Von: *sr-users <sr-users-boun...@lists.kamailio.org>
> <sr-users-boun...@lists.kamailio.org> im Auftrag von Daniel-Constantin
> Mierla <mico...@gmail.com> <mico...@gmail.com>
> *Antworten an: *"mico...@gmail.com" <mico...@gmail.com>
> <mico...@gmail.com> <mico...@gmail.com>, "Kamailio (SER) - Users Mailing
> List" <sr-users@lists.kamailio.org> <sr-users@lists.kamailio.org>
> *Datum: *Montag, 15. April 2019 um 09:07
> *An: *Aymeric Moizard <amoiz...@gmail.com> <amoiz...@gmail.com>,
> "Kamailio (SER) - Users Mailing List" <sr-users@lists.kamailio.org>
> <sr-users@lists.kamailio.org>
> *Betreff: *Re: [SR-Users] Kamailio stop to process incoming SIP traffic
> via TCP.
>
>
>
> Hello Aymeric,
>
> would you be able to test with tls module compiled against libssl 1.1 and
> using the pre-loaded shared object workaround?
>
>   *
> https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkamailio%2Fkamailio%2Ftree%2Fmaster%2Fsrc%2Fmodules%2Ftls%2Futils%2Fopenssl_mutex_shared&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526480174&sdata=d9E%2Fy4cvdLkGCPUexoCJ7tws3QL4rFqz5ebnMGnsESQ%3D&reserved=0>
>
> You should be able to use it with any version, no need to test with
> kamailio master branch.
>
> Just clone the master branch, then:
>
> cd src/modules/tls/utils/openssl_mutex_shared
>
> make
>
> Either from there or copy openssl_mutex_shared.so to a location you want,
> then pre-load it before starting your version of Kamailio.
>
> The README.md in the folder has some more details.
>
> I would like to have some validation that it works fine before approaching
> this topic with libssl project to allow to init the locks with shared
> process option.
>
> Thanks,
> Daniel
>
> On 26.03.19 16:18, Daniel-Constantin Mierla wrote:
>
> Hello,
>
> yep, locking there is expected, as listing the tls connections wait for no
> other processes to change the content of internal tls connection
> structures. So it is a side effect of libssl/libcrypto getting stuck and
> the other processing waiting for it to move one. I have the Kamailio
> training in USA these days, so the trip and schedule of the day didn't
> allow me to look more at the libsll/libcrypto code in order to find a
> solution here. It is a high priority in my list, as I get time during the
> next days.
>
> Cheers,
> Daniel
>
> On 26.03.19 15:55, Aymeric Moizard wrote:
>
> Hi All,
>
>
>
> I was debugging a TCP issue (most probably, I may start a thread for this
> question).
>
>
>
> I was trying to get some info for TCP and TLS.
>
>
>
> I typed:
>
> $> sudo kamctl rpc tls.list
>
>
>
> And waited for a while.... until... I realized that my User-Agent,
> connected with TCP was not able to register any more. I think the rpc
> command has introduced something wrong.
>
>
>
> The device can successfully "connect", send the REGISTER over the
> established TCP connection. The REGISTER do not appear in the logs any
> more, I don't see any traffic for TCP any more. So the behavior is the same
> as I had before: TCP and TLS are both not working and UDP is still working
> fine.
>
>
>
> kamctl do not work any more... so kamctl trap do not work...
>
>
>
> I have been able to type.. manually... for (all?) kamailio threads:
>
>
>
> gdb /usr/sbin/kamailio 16500 -batch --eval-command="bt full" >>
> kamailio-trap-tcp-down.txt
>
>
>
> I'm temporarly puting the backtrace I have here:
>
> https://sip.antisip.com/kamailio-trap-tcp-down.txt
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsip.antisip.com%2Fkamailio-trap-tcp-down.txt&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=1lfFxvR0m4PVcfnYsrrIO%2FM2nbGK6zfpl2C01O2c7M0%3D&reserved=0>
>
>
>
> You can see a thread stuck on the json command line: "tls_list"
>
> And many other waiting on CRYPTO_THREAD_write_lock
>
> ? might be related to: https://github.com/openssl/openssl/issues/5376
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fissues%2F5376&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=D5Fb4U3trdbRUY7ifMLSc5KE4mAxjK2%2BzOy8nSD1Rks%3D&reserved=0>
>
> SIDE NOTE:
>
> Right before I was typing the last gdb command for the last thread,
> kamailio
>
> has crashed: This was around 5 minutes after the dead lock started.
>
>
>
> Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core> [core/tcp_main.c:2561]:
> tcpconn_do_send(): failed to send on 0x7ff8dfc2fdc8 (91.121.30.149:5061->
> 62.210.97.21:49351
> <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F62.210.97.21%3A49351&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526500195&sdata=9XqEUKoMwNEvCPFtKfvB0c43yk1GcSzYOiPdY9Pj1uo%3D&reserved=0>):
> Broken pipe (32)
>
> Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core> [core/tcp_read.c:1505]:
> tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7ff8dfc2fdc8 r:
> 0x7ff8dfc2fe48 (-1)
>
> Mar 26 14:47:11 sip kamailio[16493]: WARNING: <core>
> [core/tcp_read.c:1848]: handle_io(): F_TCPCONN connection marked as bad:
> 0x7ff8dfa6a408 id 846 refcnt 3
>
> Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core> [main.c:755]:
> handle_sigs(): child process 16374 exited by a signal 11
>
> Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core> [main.c:758]:
> handle_sigs(): core was not generated
>
> Mar 26 14:47:11 sip kamailio[16371]: INFO: <core> [main.c:781]:
> handle_sigs(): terminating due to SIGCHLD
>
> Mar 26 14:47:11 sip kamailio[16493]: INFO: <core> [main.c:836]: sig_usr():
> signal 15 received
>
> Mar 26 14:47:11 sip kamailio[16500]: INFO: <core> [main.c:836]: sig_usr():
> signal 15 received
>
> Mar 26 14:47:11 sip kamailio[16479]: INFO: <core> [main.c:836]: sig_usr():
> signal 15 received
>
>
>
>
>
> Unfortunalty, even if I did my best to setup my service to generate a core
> on crash, I still have "core was not generated".... (debian stretch)
>
>
>
> Tks for reading!
>
> Regards
>
> Aymeric
>
>
>
>
>
>
>
>
-- 
Antisip - http://www.antisip.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Reply via email to