Hi, I just repeated the test and there is no Kamailio process at 100%, they are low all the time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with 100 CPS I can't put it on debug mode but only in info. Regards, Tiago Ovidiu Sas <[email protected]> escreveu no dia quinta, 23/12/2021 à(s) 03:15: > Check with htop if you don't have any kamailio procs at 100% CPU (due > to a deadlock). > Do you see any errors in the logs? > > -ovidiu > > > On Wed, Dec 22, 2021 at 11:44 AM pwerspire <[email protected]> wrote: > > > > The CPU is around 30% during the test, is always really stable and > memory doesn't increase much. > > > > I'm using children=8. > > > > > > > > Sergio Charrua <[email protected]> escreveu no dia quarta, > 22/12/2021 à(s) 16:27: > >> > >> any statistics regarding CPU use? how many threads? > >> > >> Sérgio Charrua > >> > >> > >> www.voip.pt > >> Tel.: +351 21 130 71 77 > >> > >> Email : [email protected] > >> > >> This message and any files or documents attached are strictly > confidential or otherwise legally protected. > >> > >> It is intended only for the individual or entity named. If you are not > the named addressee or have received this email in error, please inform the > sender immediately, delete it from your system and do not copy or disclose > it or its contents or use it for any purpose. Please also note that > transmission cannot be guaranteed to be secure or error-free. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <[email protected]> wrote: > >>> > >>> Hi Sergio, > >>> > >>> Thanks for your reply. > >>> > >>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the > number of simultaneous calls is stable at around 1071 at any given time. > >>> > >>> Also, I can always see INVITES going inside the server but Kamailio > stops answering them, even if you stop sending calls for a long time, and > then send a single call, Kamailio will not reply anymore to the Invites. > >>> > >>> When Kamailio stops replying to INVITES, If I do for example a "kamcmd > dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list" > it will not provide any output and not give back the prompt( before it > would just answer that the reply was too big). > >>> I will need to use "control + c" to get the prompt of the shell back > and after that, all kamcmd commands stop returning any output or giving > back the prompt. > >>> > >>> If the Kamailio is then restarted, everything starts working correctly > again. Also, another note is that this seems only to happen when DMQ is in > use and replicating to the fallback Kamailio. > >>> > >>> Regards, > >>> Tiago > >>> > >>> > >>> Sergio Charrua <[email protected]> escreveu no dia terça, > 21/12/2021 à(s) 13:38: > >>>> > >>>> What is the call length of each call? How many NICs you have on that > server? what networking switches are you using? all these variables have > huge impacts on the call flow! > >>>> > >>>> Let's say you make 100CPS and the call length is 10 minutes. > >>>> At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). > You will have around 64Mbps bandwidth being used. > >>>> At 1 minute you get 6.000 ongoing calls and will use 390Mbps > bandwidth! > >>>> Near the 10th minute you will be using almost 4Gbps bandwidth and > 60.000 ongoing calls!! > >>>> > >>>> Maybe the issue is not within Kamailio and/or the media server, but > within your network capacity ... > >>>> > >>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around > 400cps with calls up to 30seconds length, and RTPEngine goes OK too. I am > sure I could do more, but I don't bother much, this is more than enough! > >>>> But I have an enormous amount of bandwidth and 3 NICs on the server. > NICs are the key . Remember that Kamailio only processes SIP messages. > >>>> At 5 minutes, using the above call details, you've already sent > 30.000 SIP INVITES minimum and many more SIP messages during that time. You > need a very good network interface to do that kind of job! > >>>> Add 30.000 channels / RTP ports to your media server and you get a > whole lot of data going on in your network. I bet even the UTP5 cables > won't handle the heat =:o) > >>>> > >>>> Also, check the RTP port range your media server was configured with. > >>>> On Asterisk, usually ports 10.000 to 20.000 are set, which is about > 10.000 calls. In the above example, you would stop receiving calls at > 2minutes because Asterisk has all RTP ports busy/in use! > >>>> I don't know about other media servers, but I would bet this is > pretty much the same... > >>>> > >>>> Hope this helps! > >>>> > >>>> > >>>> Sérgio Charrua > >>>> > >>>> > >>>> www.voip.pt > >>>> Tel.: +351 21 130 71 77 > >>>> > >>>> Email : [email protected] > >>>> > >>>> This message and any files or documents attached are strictly > confidential or otherwise legally protected. > >>>> > >>>> It is intended only for the individual or entity named. If you are > not the named addressee or have received this email in error, please inform > the sender immediately, delete it from your system and do not copy or > disclose it or its contents or use it for any purpose. Please also note > that transmission cannot be guaranteed to be secure or error-free. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <[email protected]> > wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> I have been doing some load testing with Kamailio and having some > issues. > >>>>> > >>>>> I have been trying with 100 CPS and at the beginning, everything is > working well but after some time (it can be 10 minutes or for example 40 > minutes, is just random but normally happens more in the first 10 minutes) > Kamailio stop replying to all SIP messages but still processing HTTP > requests, some commands, etc. > >>>>> > >>>>> If I use "kamcmd dlg.list" no output happens and after that, all > kamcmd commands just keep loading with no output. > >>>>> > >>>>> Kamailio is sharing Dialogs and htables using DMQ with other > Kamailio that is the failover, if I shut down the failover one (so no DMQ > replication), the test works well with the 100 CPS. > >>>>> > >>>>> Also tried with modparam("dmq", "worker_usleep", 1000) but the > behavior is the same, it will stop processing traffic. > >>>>> > >>>>> This is some of configuration used: > >>>>> > >>>>> # ----------------- setting module-specific parameters > --------------- > >>>>> modparam("dmq", "server_address", "sip:INTERNAL_INSTANCE_IP:5060") > >>>>> modparam("dmq", "notification_address", "DMQ_NOTIFICATION_ADDRESS") > >>>>> > >>>>> modparam("dmq", "multi_notify", 1) > >>>>> modparam("dmq", "ping_interval", 10) > >>>>> modparam("dmq", "num_workers",4) > >>>>> > >>>>> > >>>>> modparam("htable", "enable_dmq", 1) > >>>>> modparam("htable", "dmq_init_sync", 1) > >>>>> > >>>>> > >>>>> # ----- dialog params ----- > >>>>> modparam("dialog", "dlg_flag", FLD_DLG) > >>>>> modparam("dialog", "dlg_match_mode", 1) > >>>>> modparam("dialog", "db_url", DBURL_RW) > >>>>> modparam("dialog", "db_mode", 0) > >>>>> modparam("dialog", "enable_dmq", 1) > >>>>> modparam("dialog", "db_update_period", 10) > >>>>> modparam("dialog", "h_id_start", H_ID_START) > >>>>> modparam("dialog", "h_id_step", H_ID_STEP) > >>>>> > >>>>> Memory: > >>>>> > >>>>> Shared: 4096 > >>>>> Private: 512 > >>>>> > >>>>> kamailio -v > >>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef > >>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, > F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, > USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, > HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED > >>>>> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE > 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > >>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > >>>>> id: 473cef > >>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1 > >>>>> > >>>>> > >>>>> In attach is the trap collected after the issue happens. > >>>>> Any more logs or configurations that can help identify or solve the > issue? > >>>>> > >>>>> Thanks for the help, > >>>>> Regards, > >>>>> Tiago > >>>>> __________________________________________________________ > >>>>> Kamailio - Users Mailing List - Non Commercial Discussions > >>>>> * [email protected] > >>>>> Important: keep the mailing list in the recipients, do not reply > only to the sender! > >>>>> Edit mailing list options or unsubscribe: > >>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > >>>> > >>>> __________________________________________________________ > >>>> Kamailio - Users Mailing List - Non Commercial Discussions > >>>> * [email protected] > >>>> Important: keep the mailing list in the recipients, do not reply only > to the sender! > >>>> Edit mailing list options or unsubscribe: > >>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > >>> > >>> __________________________________________________________ > >>> Kamailio - Users Mailing List - Non Commercial Discussions > >>> * [email protected] > >>> Important: keep the mailing list in the recipients, do not reply only > to the sender! > >>> Edit mailing list options or unsubscribe: > >>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > >> > >> __________________________________________________________ > >> Kamailio - Users Mailing List - Non Commercial Discussions > >> * [email protected] > >> Important: keep the mailing list in the recipients, do not reply only > to the sender! > >> Edit mailing list options or unsubscribe: > >> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > > __________________________________________________________ > > Kamailio - Users Mailing List - Non Commercial Discussions > > * [email protected] > > Important: keep the mailing list in the recipients, do not reply only to > the sender! > > Edit mailing list options or unsubscribe: > > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > > -- > VoIP Embedded, Inc. > http://www.voipembedded.com > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > * [email protected] > Important: keep the mailing list in the recipients, do not reply only to > the sender! > Edit mailing list options or unsubscribe: > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
