Hi Daniel, Just checking in to get any updates/findings on the stack trace I provided above.
Thanks. On Thu, Sep 19, 2019 at 11:29 AM Andrew Chen <[email protected]> wrote: > So what I pasted above is the same segfault we got from libcrypto.so > > On Thu, Sep 19, 2019 at 11:03 AM Daniel-Constantin Mierla < > [email protected]> wrote: > >> You do not need to reproduce if you still have the core files and >> installed the debug symbols for your version, just run again gdb on the >> core and get the backtraces. >> >> Cheers, >> Daniel >> On 18.09.19 22:35, Andrew Chen wrote: >> >> Yup understood. I'll provide the bt when we can reproduce this. >> >> Thanks >> >> On Wed, Sep 18, 2019 at 4:32 PM Henning Westerholt <[email protected]> >> wrote: >> >>> Hi Andrew, >>> >>> the 5.2.4 does not contain the 5.3-pre1 workaround yet. >>> >>> So if you test the workaround you need to install the lastest 5.3 from >>> the git master debian repository. >>> >>> But nevertheless it would be good to look at the backtrace, while you >>> have it. >>> >>> Cheers, >>> >>> Henning >>> Am 18.09.19 um 22:10 schrieb Joel Serrano: >>> >>> Hi Andrew, >>> >>> Download from here: >>> http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/ >>> >>> You have kamailio-dbg v5.2.4 (latest 5.2) for all distros... >>> >>> Example for amd64+stretch: >>> http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/kamailio-dbg_5.2.4%2Bstretch_amd64.deb >>> >>> Cheers, >>> Joel. >>> >>> >>> >>> >>> >>> On Wed, Sep 18, 2019 at 12:16 PM Andrew Chen <[email protected]> wrote: >>> >>>> Hi Henning, >>>> >>>> I'm running 5.2.4. Will this deb package work? >>>> >>>> On Wed, Sep 18, 2019 at 2:39 PM Andrew Chen <[email protected]> wrote: >>>> >>>>> Thanks Henning >>>>> >>>>> On Wed, Sep 18, 2019 at 2:13 PM Henning Westerholt <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello Andrew, >>>>>> >>>>>> strange - it should be there, as it is available on deb.kamailio.org. >>>>>> >>>>>> >>>>>> Here e.g. the one for 64bit stretch: >>>>>> >>>>>> >>>>>> https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-dbg_5.3.0~pre1%2B0~20190918005413.1490%2Bstretch_amd64.deb >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Henning >>>>>> Am 18.09.19 um 18:52 schrieb Andrew Chen: >>>>>> >>>>>> I don't see one for debug: >>>>>> >>>>>> root@ashintgkama51:~ # apt-cache search kamailio >>>>>> >>>>>> resource-agents - Cluster Resource Agents >>>>>> >>>>>> kamailio - very fast, dynamic and configurable SIP server >>>>>> >>>>>> kamailio-autheph-modules - authentication using ephemeral credentials >>>>>> for the Kamailio SIP server >>>>>> >>>>>> kamailio-berkeley-bin - Berkeley database module for Kamailio - >>>>>> helper program >>>>>> >>>>>> kamailio-berkeley-modules - Berkeley database module for the Kamailio >>>>>> SIP server >>>>>> >>>>>> kamailio-carrierroute-modules - carrierroute module for the Kamailio >>>>>> SIP server >>>>>> >>>>>> kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server >>>>>> >>>>>> kamailio-cpl-modules - CPL module (CPL interpreter engine) for the >>>>>> Kamailio SIP server >>>>>> >>>>>> kamailio-erlang-modules - Erlang modules for the Kamailio SIP server >>>>>> >>>>>> kamailio-geoip-modules - geoip module for the Kamailio SIP Server >>>>>> >>>>>> kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server >>>>>> >>>>>> kamailio-ims-modules - IMS module for the Kamailio SIP server >>>>>> >>>>>> kamailio-json-modules - Json parser and Json-RPC modules for the >>>>>> Kamailio SIP server >>>>>> >>>>>> On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> You have to install the debug symbols package, otherwise the >>>>>>> backtrace doesn't show any useful information. It just require to >>>>>>> install >>>>>>> that package and then get again the backtraces. The package name should >>>>>>> be >>>>>>> kamailio-dbg. >>>>>>> >>>>>>> Cheers, >>>>>>> Daniel >>>>>>> On 18.09.19 16:37, Andrew Chen wrote: >>>>>>> >>>>>>> Sorry..forgot bt full output >>>>>>> >>>>>>> (gdb) bt full >>>>>>> >>>>>>> #0 0x0000562fcc3544c0 in ?? () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #1 0x0000562fcc354632 in ?? () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #2 0x0000562fcc354974 in local_timer_run () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #3 0x0000562fcc3fb256 in ?? () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #4 0x0000562fcc3fb64c in tcp_receive_loop () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #5 0x0000562fcc2acf52 in tcp_init_children () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #6 0x0000562fcc1ad94a in main_loop () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> #7 0x0000562fcc1b4a7b in main () >>>>>>> >>>>>>> No symbol table info available. >>>>>>> >>>>>>> (gdb) >>>>>>> >>>>>>> On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen <[email protected]> wrote: >>>>>>> >>>>>>>> backtrace output: >>>>>>>> >>>>>>>> >>>>>>>> (gdb) bt >>>>>>>> >>>>>>>> #0 0x0000562fcc3544c0 in ?? () >>>>>>>> >>>>>>>> #1 0x0000562fcc354632 in ?? () >>>>>>>> >>>>>>>> #2 0x0000562fcc354974 in local_timer_run () >>>>>>>> >>>>>>>> #3 0x0000562fcc3fb256 in ?? () >>>>>>>> >>>>>>>> #4 0x0000562fcc3fb64c in tcp_receive_loop () >>>>>>>> >>>>>>>> #5 0x0000562fcc2acf52 in tcp_init_children () >>>>>>>> >>>>>>>> #6 0x0000562fcc1ad94a in main_loop () >>>>>>>> >>>>>>>> #7 0x0000562fcc1b4a7b in main () >>>>>>>> >>>>>>>> (gdb) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'll look for the core. >>>>>>>>> >>>>>>>>> As part of our release process, I don't think we can >>>>>>>>> release master branch of Kamailio. >>>>>>>>> >>>>>>>>> On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> did you get core files? If yes, it would be good to see the gdb >>>>>>>>>> bracktraces. >>>>>>>>>> >>>>>>>>>> Have you used the pre-loaded library workaround? >>>>>>>>>> >>>>>>>>>> The code was not backported to branch 5.2 yet, I wanted to wait a >>>>>>>>>> bit and see if there are reports with good/bad results. If you want >>>>>>>>>> to use >>>>>>>>>> it, then you have to install nightly builds from master branch >>>>>>>>>> (development >>>>>>>>>> version, should be 5.3.0-pre1 at this moment). >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Daniel >>>>>>>>>> On 18.09.19 15:26, Andrew Chen wrote: >>>>>>>>>> >>>>>>>>>> Hi Daniel, >>>>>>>>>> Which apt repository can we get this workaround? Today I >>>>>>>>>> noticed our Kamailio 5.2.4 cored during one of our load test >>>>>>>>>> sessions: >>>>>>>>>> >>>>>>>>>> Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at >>>>>>>>>> 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f8501dd4000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at >>>>>>>>>> 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f1a8f90f000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip >>>>>>>>>> 00007f0a85dac128 sp 00007fff6847aad0 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f0a85c40000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip >>>>>>>>>> 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f0a85c40000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at >>>>>>>>>> 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f75e1136000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at >>>>>>>>>> 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f0f37ead000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at >>>>>>>>>> 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f8043b0a000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at >>>>>>>>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7ff59d250000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at >>>>>>>>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7ff59d250000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at >>>>>>>>>> 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f22dd4db000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at >>>>>>>>>> 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f7ccae9b000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at >>>>>>>>>> 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7fb309baf000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip >>>>>>>>>> 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7fc5908d7000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at >>>>>>>>>> 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in >>>>>>>>>> libcrypto.so.1.1[7fcc238c6000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip >>>>>>>>>> 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in >>>>>>>>>> libcrypto.so.1.1[7fcc238c6000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at >>>>>>>>>> 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in >>>>>>>>>> libcrypto.so.1.1[7f586f78e000+29b000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip >>>>>>>>>> 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in >>>>>>>>>> kamailio[562fcc17c000+4a2000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip >>>>>>>>>> 00007f96c3735c3e sp 00007ffe825ca270 error 4 in >>>>>>>>>> dialog.so[7f96c36d6000+be000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip >>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>>>>>>>> kamailio[562fcc17c000+4a2000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip >>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in >>>>>>>>>> tm.so[7f96c5da5000+13f000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip >>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>>>>>>>> kamailio[562fcc17c000+4a2000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip >>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>>>>>>>> kamailio[562fcc17c000+4a2000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip >>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in >>>>>>>>>> tm.so[7f96c5da5000+13f000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip >>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in >>>>>>>>>> tm.so[7f96c5da5000+13f000] >>>>>>>>>> >>>>>>>>>> [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip >>>>>>>>>> 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in >>>>>>>>>> kamailio[562fcc17c000+4a2000] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and I was wondering if this core is related. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Daniel, >>>>>>>>>>> >>>>>>>>>>> Thanks a lot for this work; it's definitely important that >>>>>>>>>>> Kamailio >>>>>>>>>>> build against OpenSSL v1.1, and I'll be happy to contribute some >>>>>>>>>>> testing >>>>>>>>>>> in a live environment in the coming weeks. Much appreciated. >>>>>>>>>>> >>>>>>>>>>> Have a good weekend, >>>>>>>>>>> >>>>>>>>>>> -- Alex >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin >>>>>>>>>>> Mierla wrote: >>>>>>>>>>> >>>>>>>>>>> > Hello, >>>>>>>>>>> > after couple of months waiting for feedback about the >>>>>>>>>>> workaround with the >>>>>>>>>>> > preloaded shared library to set the PTHREAD_PROCESS_SHARED >>>>>>>>>>> attribute for >>>>>>>>>>> > pthread mutex/rwlock in order to avoid deadlocks and as it >>>>>>>>>>> seems to be the >>>>>>>>>>> > solution, I just pushed a few commits to move that code in the >>>>>>>>>>> core of >>>>>>>>>>> > kamailio. >>>>>>>>>>> > >>>>>>>>>>> > It this way the initialization with the PTHREAD_PROCESS_SHARED >>>>>>>>>>> attribute is >>>>>>>>>>> > done by default when libssl v1.1 is detected at compile time, >>>>>>>>>>> without the >>>>>>>>>>> > need of using the approach with a preloaded library. >>>>>>>>>>> > >>>>>>>>>>> > The shared library is still kept in the tls module: >>>>>>>>>>> > * >>>>>>>>>>> > >>>>>>>>>>> https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared >>>>>>>>>>> > >>>>>>>>>>> > Probably is going to be removed in the near future. >>>>>>>>>>> > >>>>>>>>>>> > I am asking the community members that use Kamailio on systems >>>>>>>>>>> with libssl >>>>>>>>>>> > v1.1+ to test the master branch and give feedback. The master >>>>>>>>>>> branch is >>>>>>>>>>> > anyhow in testing phase for preparing the next major release >>>>>>>>>>> v5.3.0, so it >>>>>>>>>>> > will help for that as well. >>>>>>>>>>> > >>>>>>>>>>> > If no issue is going to be reported soon, the commits will be >>>>>>>>>>> backported to >>>>>>>>>>> > stable branches, to re-enable smooth use of Kamailio with >>>>>>>>>>> libssl v1.1+. >>>>>>>>>>> > >>>>>>>>>>> > When libssl v1.1+ is detected, the core links with pthread, >>>>>>>>>>> but that was >>>>>>>>>>> > already present in the Makefiles.defs for different compile >>>>>>>>>>> options. The >>>>>>>>>>> > pthread more or less a default library on linuxes/unixes, so >>>>>>>>>>> this should >>>>>>>>>>> > not introduce any major dependency to the Kamailio core. >>>>>>>>>>> > >>>>>>>>>>> > Cheers, >>>>>>>>>>> > Daniel >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > Daniel-Constantin Mierla - http://www.asipto.com >>>>>>>>>>> > http://twitter.com/#!/miconda - >>>>>>>>>>> http://www.linkedin.com/in/miconda >>>>>>>>>>> >>>>>>>>>>> > _______________________________________________ >>>>>>>>>>> > Kamailio (SER) - Users Mailing List >>>>>>>>>>> > [email protected] >>>>>>>>>>> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Alex Balashov | Principal | Evariste Systems LLC >>>>>>>>>>> >>>>>>>>>>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) >>>>>>>>>>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Kamailio (SER) - Users Mailing List >>>>>>>>>>> [email protected] >>>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Andy Chen >>>>>>>>>> Sr. Telephony Lead Engineer >>>>>>>>>> 415 516 5535 (M) >>>>>>>>>> achen@ <[email protected]>fuze.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Confidentiality Notice: The information contained in this e-mail >>>>>>>>>> and any >>>>>>>>>> attachments may be confidential. If you are not an intended >>>>>>>>>> recipient, you >>>>>>>>>> are hereby notified that any dissemination, distribution or >>>>>>>>>> copying of this >>>>>>>>>> e-mail is strictly prohibited. If you have received this e-mail >>>>>>>>>> in error, >>>>>>>>>> please notify the sender and permanently delete the e-mail and any >>>>>>>>>> attachments immediately. You should not retain, copy or use this >>>>>>>>>> e-mail or >>>>>>>>>> any attachment for any purpose, nor disclose all or any part of >>>>>>>>>> the >>>>>>>>>> contents to any other person. Thank you.* >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Kamailio (SER) - Users Mailing >>>>>>>>>> [email protected]https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >>>>>>>>>> www.linkedin.com/in/miconda >>>>>>>>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- >>>>>>>>>> https://asipto.com/u/kat >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Andy Chen >>>>>>>>> Sr. Telephony Lead Engineer >>>>>>>>> 415 516 5535 (M) >>>>>>>>> achen@ <[email protected]>fuze.com >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Andy Chen >>>>>>>> Sr. Telephony Lead Engineer >>>>>>>> 415 516 5535 (M) >>>>>>>> achen@ <[email protected]>fuze.com >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Andy Chen >>>>>>> Sr. Telephony Lead Engineer >>>>>>> 415 516 5535 (M) >>>>>>> achen@ <[email protected]>fuze.com >>>>>>> >>>>>>> >>>>>>> *Confidentiality Notice: The information contained in this e-mail >>>>>>> and any >>>>>>> attachments may be confidential. If you are not an intended >>>>>>> recipient, you >>>>>>> are hereby notified that any dissemination, distribution or copying >>>>>>> of this >>>>>>> e-mail is strictly prohibited. If you have received this e-mail in >>>>>>> error, >>>>>>> please notify the sender and permanently delete the e-mail and any >>>>>>> attachments immediately. You should not retain, copy or use this >>>>>>> e-mail or >>>>>>> any attachment for any purpose, nor disclose all or any part of the >>>>>>> contents to any other person. Thank you.* >>>>>>> >>>>>>> -- >>>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >>>>>>> www.linkedin.com/in/miconda >>>>>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- >>>>>>> https://asipto.com/u/kat >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Andy Chen >>>>>> Sr. Telephony Lead Engineer >>>>>> 415 516 5535 (M) >>>>>> achen@ <[email protected]>fuze.com >>>>>> >>>>>> >>>>>> *Confidentiality Notice: The information contained in this e-mail and >>>>>> any >>>>>> attachments may be confidential. If you are not an intended >>>>>> recipient, you >>>>>> are hereby notified that any dissemination, distribution or copying >>>>>> of this >>>>>> e-mail is strictly prohibited. If you have received this e-mail in >>>>>> error, >>>>>> please notify the sender and permanently delete the e-mail and any >>>>>> attachments immediately. You should not retain, copy or use this >>>>>> e-mail or >>>>>> any attachment for any purpose, nor disclose all or any part of the >>>>>> contents to any other person. Thank you.* >>>>>> >>>>>> _______________________________________________ >>>>>> Kamailio (SER) - Users Mailing >>>>>> [email protected]https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>> >>>>>> -- >>>>>> Kamailio Merchandising - https://skalatan.de/merchandising/ >>>>>> Kamailio services - https://skalatan.de/services >>>>>> Henning Westerholt - https://skalatan.de/blog/ >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Andy Chen >>>>> Sr. Telephony Lead Engineer >>>>> 415 516 5535 (M) >>>>> achen@ <[email protected]>fuze.com >>>>> >>>>> >>>> >>>> -- >>>> Andy Chen >>>> Sr. Telephony Lead Engineer >>>> 415 516 5535 (M) >>>> achen@ <[email protected]>fuze.com >>>> >>>> >>>> *Confidentiality Notice: The information contained in this e-mail and >>>> any >>>> attachments may be confidential. If you are not an intended recipient, >>>> you >>>> are hereby notified that any dissemination, distribution or copying of >>>> this >>>> e-mail is strictly prohibited. If you have received this e-mail in >>>> error, >>>> please notify the sender and permanently delete the e-mail and any >>>> attachments immediately. You should not retain, copy or use this e-mail >>>> or >>>> any attachment for any purpose, nor disclose all or any part of the >>>> contents to any other person. Thank you.* >>>> _______________________________________________ >>>> Kamailio (SER) - Users Mailing List >>>> [email protected] >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>> -- >>> Kamailio Merchandising - https://skalatan.de/merchandising/ >>> Kamailio services - https://skalatan.de/services >>> Henning Westerholt - https://skalatan.de/blog/ >>> >>> >> >> -- >> Andy Chen >> Sr. Telephony Lead Engineer >> 415 516 5535 (M) >> achen@ <[email protected]>fuze.com >> >> >> *Confidentiality Notice: The information contained in this e-mail and any >> attachments may be confidential. If you are not an intended recipient, you >> are hereby notified that any dissemination, distribution or copying of >> this >> e-mail is strictly prohibited. If you have received this e-mail in error, >> please notify the sender and permanently delete the e-mail and any >> attachments immediately. You should not retain, copy or use this e-mail or >> any attachment for any purpose, nor disclose all or any part of the >> contents to any other person. Thank you.* >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing >> [email protected]https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- >> www.linkedin.com/in/miconda >> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- >> https://asipto.com/u/kat >> >> > > -- > Andy Chen > Sr. Telephony Lead Engineer > 415 516 5535 (M) > achen@ <[email protected]>fuze.com > > -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <[email protected]>fuze.com -- *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
