Check with htop if you don't have any kamailio procs at 100% CPU (due to a deadlock). Do you see any errors in the logs?
-ovidiu On Wed, Dec 22, 2021 at 11:44 AM pwerspire <[email protected]> wrote: > > The CPU is around 30% during the test, is always really stable and memory > doesn't increase much. > > I'm using children=8. > > > > Sergio Charrua <[email protected]> escreveu no dia quarta, 22/12/2021 > à(s) 16:27: >> >> any statistics regarding CPU use? how many threads? >> >> Sérgio Charrua >> >> >> www.voip.pt >> Tel.: +351 21 130 71 77 >> >> Email : [email protected] >> >> This message and any files or documents attached are strictly confidential >> or otherwise legally protected. >> >> It is intended only for the individual or entity named. If you are not the >> named addressee or have received this email in error, please inform the >> sender immediately, delete it from your system and do not copy or disclose >> it or its contents or use it for any purpose. Please also note that >> transmission cannot be guaranteed to be secure or error-free. >> >> >> >> >> >> >> >> >> >> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <[email protected]> wrote: >>> >>> Hi Sergio, >>> >>> Thanks for your reply. >>> >>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number of >>> simultaneous calls is stable at around 1071 at any given time. >>> >>> Also, I can always see INVITES going inside the server but Kamailio stops >>> answering them, even if you stop sending calls for a long time, and then >>> send a single call, Kamailio will not reply anymore to the Invites. >>> >>> When Kamailio stops replying to INVITES, If I do for example a "kamcmd >>> dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list" >>> it will not provide any output and not give back the prompt( before it >>> would just answer that the reply was too big). >>> I will need to use "control + c" to get the prompt of the shell back and >>> after that, all kamcmd commands stop returning any output or giving back >>> the prompt. >>> >>> If the Kamailio is then restarted, everything starts working correctly >>> again. Also, another note is that this seems only to happen when DMQ is in >>> use and replicating to the fallback Kamailio. >>> >>> Regards, >>> Tiago >>> >>> >>> Sergio Charrua <[email protected]> escreveu no dia terça, 21/12/2021 >>> à(s) 13:38: >>>> >>>> What is the call length of each call? How many NICs you have on that >>>> server? what networking switches are you using? all these variables have >>>> huge impacts on the call flow! >>>> >>>> Let's say you make 100CPS and the call length is 10 minutes. >>>> At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You >>>> will have around 64Mbps bandwidth being used. >>>> At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth! >>>> Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000 >>>> ongoing calls!! >>>> >>>> Maybe the issue is not within Kamailio and/or the media server, but within >>>> your network capacity ... >>>> >>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps >>>> with calls up to 30seconds length, and RTPEngine goes OK too. I am sure I >>>> could do more, but I don't bother much, this is more than enough! >>>> But I have an enormous amount of bandwidth and 3 NICs on the server. NICs >>>> are the key . Remember that Kamailio only processes SIP messages. >>>> At 5 minutes, using the above call details, you've already sent 30.000 SIP >>>> INVITES minimum and many more SIP messages during that time. You need a >>>> very good network interface to do that kind of job! >>>> Add 30.000 channels / RTP ports to your media server and you get a whole >>>> lot of data going on in your network. I bet even the UTP5 cables won't >>>> handle the heat =:o) >>>> >>>> Also, check the RTP port range your media server was configured with. >>>> On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000 >>>> calls. In the above example, you would stop receiving calls at 2minutes >>>> because Asterisk has all RTP ports busy/in use! >>>> I don't know about other media servers, but I would bet this is pretty >>>> much the same... >>>> >>>> Hope this helps! >>>> >>>> >>>> Sérgio Charrua >>>> >>>> >>>> www.voip.pt >>>> Tel.: +351 21 130 71 77 >>>> >>>> Email : [email protected] >>>> >>>> This message and any files or documents attached are strictly confidential >>>> or otherwise legally protected. >>>> >>>> It is intended only for the individual or entity named. If you are not the >>>> named addressee or have received this email in error, please inform the >>>> sender immediately, delete it from your system and do not copy or disclose >>>> it or its contents or use it for any purpose. Please also note that >>>> transmission cannot be guaranteed to be secure or error-free. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have been doing some load testing with Kamailio and having some issues. >>>>> >>>>> I have been trying with 100 CPS and at the beginning, everything is >>>>> working well but after some time (it can be 10 minutes or for example 40 >>>>> minutes, is just random but normally happens more in the first 10 >>>>> minutes) Kamailio stop replying to all SIP messages but still processing >>>>> HTTP requests, some commands, etc. >>>>> >>>>> If I use "kamcmd dlg.list" no output happens and after that, all kamcmd >>>>> commands just keep loading with no output. >>>>> >>>>> Kamailio is sharing Dialogs and htables using DMQ with other Kamailio >>>>> that is the failover, if I shut down the failover one (so no DMQ >>>>> replication), the test works well with the 100 CPS. >>>>> >>>>> Also tried with modparam("dmq", "worker_usleep", 1000) but the behavior >>>>> is the same, it will stop processing traffic. >>>>> >>>>> This is some of configuration used: >>>>> >>>>> # ----------------- setting module-specific parameters --------------- >>>>> modparam("dmq", "server_address", "sip:INTERNAL_INSTANCE_IP:5060") >>>>> modparam("dmq", "notification_address", "DMQ_NOTIFICATION_ADDRESS") >>>>> >>>>> modparam("dmq", "multi_notify", 1) >>>>> modparam("dmq", "ping_interval", 10) >>>>> modparam("dmq", "num_workers",4) >>>>> >>>>> >>>>> modparam("htable", "enable_dmq", 1) >>>>> modparam("htable", "dmq_init_sync", 1) >>>>> >>>>> >>>>> # ----- dialog params ----- >>>>> modparam("dialog", "dlg_flag", FLD_DLG) >>>>> modparam("dialog", "dlg_match_mode", 1) >>>>> modparam("dialog", "db_url", DBURL_RW) >>>>> modparam("dialog", "db_mode", 0) >>>>> modparam("dialog", "enable_dmq", 1) >>>>> modparam("dialog", "db_update_period", 10) >>>>> modparam("dialog", "h_id_start", H_ID_START) >>>>> modparam("dialog", "h_id_step", H_ID_STEP) >>>>> >>>>> Memory: >>>>> >>>>> Shared: 4096 >>>>> Private: 512 >>>>> >>>>> kamailio -v >>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef >>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >>>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, >>>>> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, >>>>> HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED >>>>> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, >>>>> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>>> id: 473cef >>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1 >>>>> >>>>> >>>>> In attach is the trap collected after the issue happens. >>>>> Any more logs or configurations that can help identify or solve the issue? >>>>> >>>>> Thanks for the help, >>>>> Regards, >>>>> Tiago >>>>> __________________________________________________________ >>>>> Kamailio - Users Mailing List - Non Commercial Discussions >>>>> * [email protected] >>>>> Important: keep the mailing list in the recipients, do not reply only to >>>>> the sender! >>>>> Edit mailing list options or unsubscribe: >>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> __________________________________________________________ >>>> Kamailio - Users Mailing List - Non Commercial Discussions >>>> * [email protected] >>>> Important: keep the mailing list in the recipients, do not reply only to >>>> the sender! >>>> Edit mailing list options or unsubscribe: >>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>> >>> __________________________________________________________ >>> Kamailio - Users Mailing List - Non Commercial Discussions >>> * [email protected] >>> Important: keep the mailing list in the recipients, do not reply only to >>> the sender! >>> Edit mailing list options or unsubscribe: >>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> * [email protected] >> Important: keep the mailing list in the recipients, do not reply only to the >> sender! >> Edit mailing list options or unsubscribe: >> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > * [email protected] > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
