Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com mailto:sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com mailto:o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda http://twitter.com/#%21/miconda - http://www.linkedin.com/in/miconda -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Hi All, We have solved this issue now. The problem was the wrong interpretation of the Kamailio Makefile.defs makefile; The ARCH script variable was set to a wrong string by our higher level makefile. Ea; for powerPC, Kamilio.defs expects 'ppc' as the value while the higher level makefile sets the ARCH variable to 'powerpc'. So Kamailio was compiled with wrong and missing GCC arguments. Also some C-defines were not set correctly which resulted in linking wrong 'locking' routines, like atomic_unknown.h. Instead atomic_ppc.h should be linked/compiled. Thanks for your support anyway. Best regards, Orhan Yilmaz On Fri, Nov 29, 2013 at 3:02 PM, Daniel-Constantin Mierla mico...@gmail.com wrote: Attaching with gdb and getting the backtrace (as Ovidiu said), will be very useful The function futex_wait_queue_me() is from kernel, so it might be an operation from the kernel about reading the udp packages. So we cannot tell without the full trace if a kamailio function is blocking the futex or is other standard library function. I wanted to mention also that poll methods are for tcp only, for udp kamailio uses recvfrom(). Can you also provide the output for 'kamailio -I' (that's upper case i)? Cheers, Daniel On 11/27/13 4:44 PM, Ovidiu Sas wrote: Try to attach gdb to the kamailio processes and run a full backtrace. Regards, Ovidiu Sas On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development sotas...@gmail.com wrote: Hi, In the mean time we have gathered more information on this problem: As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running. NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw0 0 (null):255 (null):*255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl Kamailio is started with the following options = -m 4 -n 3 -f cfg -D Other relevant info: - When Kamailio hangs, I also noticed that the flag inuse_transactions has always the value of '1'. Readout with kamctl monitor. - A simple cat to /proc/kamailio_pid/wchan gives us the function: futex_wait_queue_me. - All possible polling methods are used with -W parameter (sigio_rt, poll, select etc) during these tests. Non of these options did solve this problem. I hope the additional info will clarify more. Thanks in advance. Best regards, Orhan Yilmaz On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas o...@voipembedded.com wrote: In a previous e-mail, you posted a warning that you had while compiling: no native memory barrier implementations, falling back to slow lock based workarround which means that you are already running without atomic locks. Regards, Ovidiu Sas On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotas...@gmail.com wrote: Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas o...@voipembedded.com wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Hi, In the mean time we have gathered more information on this problem: As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running. NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):*8416/kamailio raw0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl Kamailio is started with the following options = -m 4 -n 3 -f cfg -D Other relevant info: - When Kamailio hangs, I also noticed that the flag inuse_transactions has always the value of '1'. Readout with kamctl monitor. - A simple cat to /proc/kamailio_pid/wchan gives us the function: futex_wait_queue_me. - All possible polling methods are used with -W parameter (sigio_rt, poll, select etc) during these tests. Non of these options did solve this problem. I hope the additional info will clarify more. Thanks in advance. Best regards, Orhan Yilmaz On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas o...@voipembedded.com wrote: In a previous e-mail, you posted a warning that you had while compiling: no native memory barrier implementations, falling back to slow lock based workarround which means that you are already running without atomic locks. Regards, Ovidiu Sas On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotas...@gmail.com wrote: Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas o...@voipembedded.com wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Try to attach gdb to the kamailio processes and run a full backtrace. Regards, Ovidiu Sas On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development sotas...@gmail.com wrote: Hi, In the mean time we have gathered more information on this problem: As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running. NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw0 0 (null):255 (null):*255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl Kamailio is started with the following options = -m 4 -n 3 -f cfg -D Other relevant info: - When Kamailio hangs, I also noticed that the flag inuse_transactions has always the value of '1'. Readout with kamctl monitor. - A simple cat to /proc/kamailio_pid/wchan gives us the function: futex_wait_queue_me. - All possible polling methods are used with -W parameter (sigio_rt, poll, select etc) during these tests. Non of these options did solve this problem. I hope the additional info will clarify more. Thanks in advance. Best regards, Orhan Yilmaz On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas o...@voipembedded.com wrote: In a previous e-mail, you posted a warning that you had while compiling: no native memory barrier implementations, falling back to slow lock based workarround which means that you are already running without atomic locks. Regards, Ovidiu Sas On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotas...@gmail.com wrote: Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas o...@voipembedded.com wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas o...@voipembedded.com wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
In a previous e-mail, you posted a warning that you had while compiling: no native memory barrier implementations, falling back to slow lock based workarround which means that you are already running without atomic locks. Regards, Ovidiu Sas On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotas...@gmail.com wrote: Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas o...@voipembedded.com wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com -- Forwarded message -- From: Sotas Development sotas...@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List sr-users@lists.sip-router.org Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas o...@voipembedded.com wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Daniel, can you add an xlog() at the start of the main route block and log a message for any request received? [...] You can put another xlog before the save() function to see if registration requests are getting there. [...] You can switch to kamailio flavour modules and see if reproduces. Thanks for the help! We switched to kamailio flavour, added some xlog messages and managed to reproduce. See below for some logging. To our surprise, the last message handled was an INVITE. We'll add more logging to see whether it is handled successfully or gets stuck somewhere. Those numbers at the start of the log message, are these child IDs? At first they alternate, but apparently child 1 and 2 stop running early on (each with an INVITE as last message). Regards, Michiel Veldkamp 0(304) WARNING: core [socket_info.c:1392]: WARNING: fix_hostname: could not rev. resolve 192.168.10.1 0(304) INFO: core [tcp_main.c:4832]: init_tcp: using epoll_lt as the io watch method (auto detected) 0(306) INFO: usrloc [hslot.c:53]: locks array size 512 0(306) INFO: core [udp_server.c:179]: INFO: udp_init: SO_RCVBUF is initially 108544 0(306) INFO: core [udp_server.c:230]: INFO: udp_init: SO_RCVBUF is finally 217088 7(315) INFO: ctl [io_listener.c:225]: io_listen_loop: using epoll_lt io watch method (config) 1(309) ERROR: script: request_route start -- method=REGISTER 2(310) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: route[REGISTRAR] start 3(311) ERROR: script: route[REGISTRAR] saving location... 2(310) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: route[REGISTRAR] start 3(311) ERROR: script: route[REGISTRAR] saving location... 3(311) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: request_route start -- method=REGISTER 1(309) ERROR: script: request_route start -- method=REGISTER 1(309) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: route[REGISTRAR] saving location... 2(310) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: request_route start -- method=INVITE 3(311) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: request_route start -- method=INVITE 1(309) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=INVITE 2(310) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: request_route start -- method=INVITE 1(309) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=INVITE 2(310) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=966002447;to_tag=3a8473f2;call_id= 331189810@192.168.10.2 ;code=200;reason=OK;src_user=Radio1_device;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1718943109;to_tag=22a97aa7;call_id= 956233863@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2 3(311) ERROR: script: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1709043140;to_tag=4aa972a8;call_id= 780056350@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1506064927;to_tag=5a73a3fc;call_id= 1129604628@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1200029211;to_tag=14c6d7d4;call_id= 1214006622@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 1(309) ERROR: script: request_route start -- method=ACK 2(310) ERROR: script: request_route start -- method=ACK 3(311) ERROR: script: request_route start -- method=ACK 2(310) ERROR: script:
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Hello Michiel , You should check with top to see the status of all kamailio processes. If you see processes that are using 100% CPU, you may have a deadlock. Connect with gdb to the process to investigate what's going on. Also, you are running on arm. How did you compiled kamailio: native or cross. When you run kamailio on an embedded system, you need to check if you have enough memory. If you start running out of memory, the OS may start killing processes randomly (check your OS logs). You definitely need to look at this issue from a broader prospective as it's not a regular x86 deployment with plenty of CPU and memory. Also tuning the config (by removing everything that you don't need might help with memory utilization). Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com On Thu, Jan 17, 2013 at 8:41 AM, Sotas Development sotas...@gmail.com wrote: Daniel, can you add an xlog() at the start of the main route block and log a message for any request received? [...] You can put another xlog before the save() function to see if registration requests are getting there. [...] You can switch to kamailio flavour modules and see if reproduces. Thanks for the help! We switched to kamailio flavour, added some xlog messages and managed to reproduce. See below for some logging. To our surprise, the last message handled was an INVITE. We'll add more logging to see whether it is handled successfully or gets stuck somewhere. Those numbers at the start of the log message, are these child IDs? At first they alternate, but apparently child 1 and 2 stop running early on (each with an INVITE as last message). Regards, Michiel Veldkamp 0(304) WARNING: core [socket_info.c:1392]: WARNING: fix_hostname: could not rev. resolve 192.168.10.1 0(304) INFO: core [tcp_main.c:4832]: init_tcp: using epoll_lt as the io watch method (auto detected) 0(306) INFO: usrloc [hslot.c:53]: locks array size 512 0(306) INFO: core [udp_server.c:179]: INFO: udp_init: SO_RCVBUF is initially 108544 0(306) INFO: core [udp_server.c:230]: INFO: udp_init: SO_RCVBUF is finally 217088 7(315) INFO: ctl [io_listener.c:225]: io_listen_loop: using epoll_lt io watch method (config) 1(309) ERROR: script: request_route start -- method=REGISTER 2(310) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: route[REGISTRAR] start 3(311) ERROR: script: route[REGISTRAR] saving location... 2(310) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: request_route start -- method=REGISTER 3(311) ERROR: script: route[REGISTRAR] start 3(311) ERROR: script: route[REGISTRAR] saving location... 3(311) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: request_route start -- method=REGISTER 1(309) ERROR: script: request_route start -- method=REGISTER 1(309) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: route[REGISTRAR] saving location... 2(310) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: route[REGISTRAR] saving location... 1(309) ERROR: script: route[REGISTRAR] saving location... done 2(310) ERROR: script: route[REGISTRAR] saving location... done 3(311) ERROR: script: request_route start -- method=INVITE 3(311) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: request_route start -- method=INVITE 1(309) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=INVITE 2(310) ERROR: script: route[REGISTRAR] start 1(309) ERROR: script: request_route start -- method=INVITE 1(309) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=INVITE 2(310) ERROR: script: route[REGISTRAR] start 2(310) ERROR: script: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=966002447;to_tag=3a8473f2;call_id=331189810@192.168.10.2;code=200;reason=OK;src_user=Radio1_device;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1718943109;to_tag=22a97aa7;call_id=956233863@192.168.10.2;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2 3(311) ERROR: script: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
Hi Ovidiu, Good points. We're not running out of memory, and all processes keep running. Top shows 0% or 1% CPU load for the Kamailio processes. We cross-compile with the CodeSourcery toolchain. The default build options drag in the file atomic_unknown.h, that produces the following compile warning: no native memory barrier implementations, falling back to slow lock based workarround Currently we're testing a version compiled with -DNOSMP (no atomic_unknown.h) that will run over the weekend. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 3:20 PM, Ovidiu Sas o...@voipembedded.com wrote: Hello Michiel , You should check with top to see the status of all kamailio processes. If you see processes that are using 100% CPU, you may have a deadlock. Connect with gdb to the process to investigate what's going on. Also, you are running on arm. How did you compiled kamailio: native or cross. When you run kamailio on an embedded system, you need to check if you have enough memory. If you start running out of memory, the OS may start killing processes randomly (check your OS logs). You definitely need to look at this issue from a broader prospective as it's not a regular x86 deployment with plenty of CPU and memory. Also tuning the config (by removing everything that you don't need might help with memory utilization). Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com On Thu, Jan 17, 2013 at 12:43 PM, Sotas Development sotas...@gmail.com wrote: Hi Ovidiu, Good points. We're not running out of memory, and all processes keep running. Top shows 0% or 1% CPU load for the Kamailio processes. We cross-compile with the CodeSourcery toolchain. The default build options drag in the file atomic_unknown.h, that produces the following compile warning: no native memory barrier implementations, falling back to slow lock based workarround Currently we're testing a version compiled with -DNOSMP (no atomic_unknown.h) that will run over the weekend. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 3:20 PM, Ovidiu Sas o...@voipembedded.com wrote: Hello Michiel , You should check with top to see the status of all kamailio processes. If you see processes that are using 100% CPU, you may have a deadlock. Connect with gdb to the process to investigate what's going on. Also, you are running on arm. How did you compiled kamailio: native or cross. When you run kamailio on an embedded system, you need to check if you have enough memory. If you start running out of memory, the OS may start killing processes randomly (check your OS logs). You definitely need to look at this issue from a broader prospective as it's not a regular x86 deployment with plenty of CPU and memory. Also tuning the config (by removing everything that you don't need might help with memory utilization). Regards, Ovidiu Sas ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
