On 27.11.19 18:15, Daniel Greenwald wrote: > yes tls is loaded before db_mysql, is that bad? Not, tls before any other modules potentially using libssl is the right order. > > Yes apt updates were run so could have been updates to anything. OS > version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we > last run a couple of weeks ago. > > Upon further digging I think the tls change was released a couple > weeks ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue. Cheers, Daniel > > > > On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla > <[email protected] <mailto:[email protected]>> wrote: > > The issue seems to come from libmysqlclient library, not to > kamailio handling sip traffic over tls. > > Is tls module loaded before the db_mysql module? > > Have you made other updates to the operating system? Or only the > change you mentioned about server requiring and verifying > certificates? > > What version of ubuntu do you run? > > I see that it uses libssl 1.0, so it is no relation with the > change in libssl 1.1 internal api. > > Cheers, > Daniel > > On 27.11.19 17:38, Daniel Greenwald wrote: >> Core was generated by `/usr/sbin/kamailio -P >> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 __memcmp_sse4_1 () at >> ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 >> 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file >> or directory. >> (gdb) bt >> #0 __memcmp_sse4_1 () at >> ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 >> #1 0x00007fc864920b79 in ?? () from >> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> #2 0x00007fc86492117a in lh_retrieve () from >> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> #3 0x00007fc864923651 in ?? () from >> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> #4 0x00007fc8649240de in ERR_get_state () from >> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> #5 0x00007fc864924326 in ?? () from >> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 >> #6 0x00007fc864c8a8e0 in SSL_get_error () from >> /lib/x86_64-linux-gnu/libssl.so.1.0.0 >> #7 0x00007fc843831cbc in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #8 0x00007fc84380d078 in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #9 0x00007fc84380d317 in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #10 0x00007fc84380e09c in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #11 0x00007fc843801fe7 in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #12 0x00007fc8438030d8 in ?? () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #13 0x00007fc843804e36 in mysql_real_query () from >> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 >> #14 0x00007fc843db0f76 in ?? () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so >> #15 0x00007fc8433bbb79 in ?? () from >> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 >> #16 0x00007fc8433bf95b in db_do_update () from >> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 >> #17 0x00007fc843db5a86 in db_mysql_update () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so >> #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so >> #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so >> #20 0x00007fc83f21a065 in dlg_onroute () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so >> #21 0x00007fc841017917 in run_rr_callbacks () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so >> #22 0x00007fc84100c0dd in ?? () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so >> #23 0x00007fc84100c7aa in loose_route () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so >> #24 0x00007fc84100ffb5 in ?? () from >> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so >> #25 0x0000000000480a22 in do_action () >> #26 0x000000000048db3f in run_actions () >> #27 0x000000000048e1fa in run_actions_safe () >> #28 0x0000000000450aed in rval_get_int () >> #29 0x00000000004551ed in rval_expr_eval_int () >> #30 0x00000000004804d0 in do_action () >> #31 0x000000000048db3f in run_actions () >> #32 0x000000000047d2a3 in do_action () >> #33 0x000000000048db3f in run_actions () >> #34 0x000000000048e306 in run_top_route () >> #35 0x00000000005b815c in receive_msg () >> #36 0x00000000004ab07f in udp_rcv_loop () >> #37 0x000000000042533c in main_loop () >> ---Type <return> to continue, or q <return> to quit--- >> #38 0x000000000042cb3a in main () >> >> >> On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla >> <[email protected] <mailto:[email protected]>> wrote: >> >> You got a core file, logs show that: >> >> 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: >> <core> [main.c:764]: handle_sigs(): core was generated >> >> Locate it and send the full backtrace with gdb from it. >> >> Cheers, >> Daniel >> >> On 27.11.19 16:03, Daniel Greenwald wrote: >>> I'm sorry you are correct, the version we are running is >>> 5.2.5. I've just run ulimit -c unlimited to hopefully get a >>> core dump next time since I don't see one in >>> /var/lib/systemd/coredump/. Not sure if there is anything I >>> need to do to get a coredump. >>> The log just shows: >>> >>> 2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: >>> <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 >>> 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: >>> <core> [main.c:761]: handle_sigs(): child process 12674 >>> exited by a signal 11 >>> 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: >>> <core> [main.c:764]: handle_sigs(): core was generated >>> 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD >>> 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: >>> <core> [main.c:842]: sig_usr(): signal 15 received >>> >>> On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> If you use the repo listed at http://deb.kamailio.org/ , >>> then it should be 5.2.5, and there is also option to >>> install 5.2 branch debs built nightly. >>> >>> Cheers, >>> Daniel >>> >>> On 27.11.19 15:37, Daniel Greenwald wrote: >>>> I'm using ubuntu packages, I believe 5.2.3 is latest? >>>> >>>> On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin >>>> Mierla <[email protected] <mailto:[email protected]>> >>>> wrote: >>>> >>>> Hello, >>>> >>>> that log message is irrelevant for actual crash. >>>> Did you get core files? Or other log messages >>>> before the one you pasted? >>>> >>>> You should upgrade first to latest 5.2.x, either >>>> 5.2.5 or latest branch 5.2. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 27.11.19 15:22, Daniel Greenwald wrote: >>>>> I'm seeing this in the log and kamailio is >>>>> crashing every hour or so: >>>>> CRITICAL: <core> [core/pass_fd.c:277]: >>>>> receive_fd(): EOF on 33/var/log/kamailio/kamailio.log >>>>> >>>>> Any ideas? Should I emergency upgrade to 5.3.1? >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.com >>>> <http://www.asipto.com> >>>> www.twitter.com/miconda <http://www.twitter.com/miconda> >>>> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >>>> Kamailio World Conference - April 27-29, 2020, in Berlin >>>> -- www.kamailioworld.com <http://www.kamailioworld.com> >>>> >>> -- >>> Daniel-Constantin Mierla -- www.asipto.com >>> <http://www.asipto.com> >>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >>> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >>> Kamailio World Conference - April 27-29, 2020, in Berlin -- >>> www.kamailioworld.com <http://www.kamailioworld.com> >>> >> -- >> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> >> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >> Kamailio World Conference - April 27-29, 2020, in Berlin -- >> www.kamailioworld.com <http://www.kamailioworld.com> >> > -- > Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> -- > www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Kamailio World Conference - April 27-29, 2020, in Berlin -- > www.kamailioworld.com <http://www.kamailioworld.com> > -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
