Thanks Ben! replies from HTTP server are below 200ms. Not sure if this is enough to be a bottleneck.... Regarding async_http, knowing that Kamailio is stateless, means (it is my understanding, at least) that the HTTP request needs to be synchronous. But I will give a try to async_http.
Atenciosamente / Kind Regards / Cordialement / Un saludo, *Sérgio Charrua* On Thu, Dec 19, 2024 at 3:49 PM Ben Kaufman <[email protected]> wrote: > What are your http query times like? You're using http_client, which > functions in a blocking manner: The http request is sent, and that thread > does nothing else until a response is received and then it continues. If > your request time is even moderately high try http_async_client. > > *Kaufman* > > *Senior Voice Engineer * > > > E: [email protected] > > > > > > > SIP.US Client Support: 800.566.9810 *|* SIPTRUNK Client Support: > 800.250.6510 *|* Flowroute Client Support: 855.356.9768 > [image: img] <https://www.sip.us/> > [image: img] <https://www.siptrunk.com/> > *[image: img] <https://www.flowroute.com/>* > > > ------------------------------ > *From:* Sergio Charrua via sr-users <[email protected]> > *Sent:* Thursday, December 19, 2024 9:08 AM > *To:* Kamailio (SER) - Users Mailing List <[email protected]> > *Cc:* Sergio Charrua <[email protected]> > *Subject:* [SR-Users] Kamailio not receiving packets on high CPS > > > *CAUTION:* This email originated from outside the organization. *Do not > click links or open attachments* unless you recognize the sender and know > the content is safe. > > Hello! > > for this ST/SH project, we are using Kamailio 5.8.4 in stateless mode, > making HTTP requests to a REST API (Java Spring) that will reply with a > JSON object, and then Kamailio replaces the Contact header and sends a SIP > 300 Multiple Choice reply to the SBC that sent the initial INVITE. > > Kamailio is running on a VM with 4vCPUs and 4GB RAM and 2 NICs. > On NIC ens224 there is a Virtual IP managed by Keepalived. > Kamailio is listening on port 5060 on ens224 and set with 32 children > process (tried with 8, 16, 24 and also 64, but with this value it was a lot > worst). > > We are doing load performance tests on our PreProd environment. With the > SBC we are sending 450 to 600 CAPS to Kamailio, and what we noticed is that > above 450 CAPS, and after less than 1 minute, Kamailio only replies SIP 100 > to some INVITEs. We also could figure out that Kamailio is not receiving > all the INVITES, despite having proved that ALL invites are sent to the > server. > It seems to me that, for some reason, the OS is somehow not able to > deliver *all* SIP packets to Kamailio, because: > - sngrep does capture all the SIP packets and shows the flows > - though the flow shows SIP invite coming from SBC to Kamailio Server, > there is no SIP 100 replied back to SBC, even though this is the 1st thing > kamailio does, so, for me, Kamailio did not received the SIP Invite > - picking a call with reinvites, which after some seconds, SBC cancels, if > I copy the Call-ID value and grep on kamailio's logs, I just do not find > any content! > > I have read some "network tuning" articles, including Linux Tune Network > Stack (Buffers Size) To Increase Networking Performance - nixCraft > <https://www.cyberciti.biz/faq/linux-tcp-tuning/> and Tuning Kamailio for > high throughput and performance | Evariste Systems Blog > <https://blog.evaristesys.com/2016/02/15/tuning-kamailio-for-high-throughput-and-performance/> > and > when running the following command I get: > > $ ss -4 -n -l | grep 5060 > udp UNCONN 0 0 10.242.17.125:5060 0.0.0.0:* > udp UNCONN 324224 0 10.242.17.146:5060 0.0.0.0:* > > According to the article, the 3rd column should be as near 0 as possible, > ideally 0. > > While I am waiting for some sysadmin with root permissions to be available > and modify some of the network parameters on this VM, I wonder if anyone > has some tips to share on how to solve this behaviour. I'm not sure, > either, what parameters to change or if they should be changed! The goal is > to have a minimum 750 CAPS. > > Lastly, some parts of the code: > > Kamailio.cfg: > > ### LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR > debug=1 > log_stderror=no > rundir="/tmp" > memdbg=5 > memlog=5 > log_facility=LOG_LOCAL0 > # configure the prefix for all log messages > log_prefix_mode = 1 > log_prefix="{$mt $hdr(CSeq) $ci} " > /* number of SIP routing processes */ > children=32 # ------------- ALREADY TRIED WITH 8,16,24... no results..... > /* uncomment the next line to disable TCP (default on) */ > disable_tcp=no > /* uncomment the next line to disable the auto discovery of local aliases > * based on reverse DNS on IPs (default on) */ > # auto_aliases=no > /* add local domain aliases */ > # alias="sip.mydomain.com" > > > ####### Custom Parameters ######### > > /* These parameters can be modified runtime via RPC interface > * - see the documentation of 'cfg_rpc' module. > * > * Format: group.id = value 'desc' description > * Access: $sel(cfg_get.group.id) or @cfg_get.group.id */ > > ####### Modules Section ######## > > /* set paths to location of modules */ > loadmodule "db_mysql.so" > loadmodule "db_cluster.so" > loadmodule "http_client.so" > loadmodule "jsonrpcs.so" > loadmodule "kex.so" > loadmodule "corex.so" > loadmodule "sl.so" > loadmodule "pv.so" > loadmodule "maxfwd.so" > loadmodule "textops.so" > loadmodule "xlog.so" > loadmodule "sanity.so" > loadmodule "jansson.so" > loadmodule "snmpstats.so" > loadmodule "file_out.so" > loadmodule "ctl.so" > loadmodule "permissions.so" > loadmodule "xhttp.so" > loadmodule "xhttp_rpc.so" > > ####### Other Args and Env Vars ######### > > /* listen addresses */ > include_file "listen.cfg" > include_file "db_conn.cfg" > > # ----------------- setting module-specific parameters -------------- > # --- DB Cluster params --- > # minimum requirement is to have DBURL1 defined > #!ifdef DBURL1 > modparam("db_cluster", "connection" , DBURL1) > #!endif > #!ifdef DBURL2 > modparam("db_cluster", "connection" , DBURL2) > #!endif > #!ifdef DBURL3 > modparam("db_cluster", "connection" , DBURL3) > #!endif > #!ifdef DBURL4 > modparam("db_cluster", "connection" , DBURL4) > #!endif > #!ifdef DBURL5 > modparam("db_cluster", "connection" , DBURL5) > #!endif > modparam("db_cluster", "cluster", DBCLUSTER) > > # --- Permissions params --- > modparam("permissions", "db_url", "cluster://k1") > modparam("permissions", "db_mode", 1) > #modparam("permissions", "reload_delta", 30) > > > # ----- jsonrpcs params ----- > modparam("jsonrpcs", "pretty_format", 1) > /* set the path to RPC fifo control file */ > modparam("jsonrpcs", "fifo_name", "/tmp/kamailio_rpc.fifo") > /* set the path to RPC unix socket control file */ > modparam("jsonrpcs", "dgram_socket", "/tmp/kamailio_rpc.sock") > > modparam("jsonrpcs", "transport", 0) > # ----- ctl params ----- > /* set the path to RPC unix socket control file */ > modparam("ctl", "binrpc", "unix:/tmp/kamailio_ctl") > > # ----- http_async_client params ----- > modparam("http_client", "query_result", 0) > modparam("http_client", "keep_connections", 0) > modparam("http_client", "connection_timeout",2) > modparam("http_client", "timeout_mode",2) > modparam("http_client", "config_file", > "/usr/local/etc/kamailio/http_client.cfg") > # ---- SNMP Stats params ---- > modparam("snmpstats", "sipEntityType", "proxyServer") > modparam("snmpstats", "sipEntityType", "redirectServer") > modparam("snmpstats", "sipEntityType", "other") > modparam("snmpstats", "snmpgetPath", "/usr/bin/") > modparam("snmpstats", "MsgQueueMinorThreshold", 20) > modparam("snmpstats", "MsgQueueMajorThreshold", 100) > modparam("snmpstats", "dlg_minor_threshold", 20) > modparam("snmpstats", "dlg_major_threshold", 100) > modparam("snmpstats", "snmpCommunity", "kamailio") > > # ---- File_Out params ---- > modparam("file_out", "base_folder", "/data/sabire002/log/kamailio/") > modparam("file_out", "file", > "name=stsh_requests;interval=1440;extension=.log") > modparam("file_out", "file", "name=cdr;interval=1440;extension=.log") > modparam("file_out", "file", "name=http;interval=1440;extension=.log") > ####### Routing Logic ######## > > include_file "includes/reqinit.cfg" > include_file "includes/handle_options.cfg" > include_file "includes/handle_cancel.cfg" > include_file "includes/handle_stir_shaken_stateless.cfg" > include_file "includes/handle_http_rpc.cfg" > > request_route { > $avp(START_TIME)=$utimef(%Y-%m-%d %H:%M:%S); > $avp(GROUPID) = allow_address_group($si, $sp); > > if ( $avp(GROUPID) == 100 || !allow_address_group($si, $sp) ) { > xlog("L_INFO", "INIT - $si:$sp is not in the allowed ACL Group > ID !\n"); > #sl_reply("401", "Address not allowed"); > exit; > }; > > if (is_method("ACK") ) { #&& t_check_trans() ){ > exit; > } > > if (is_method("INVITE")) { > > file_out("cdr","$rm|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d > %H:%M:%S)"); > send_reply("100","Trying"); > } > > route(HANDLE_OPTIONS); > > route(REQINIT); > > xlog("L_INFO"," ********** Route START ***********"); > # log the basic info regarding this call > xlog("L_INFO","=================================================== \n"); > xlog("L_INFO","New SIP message $rm with call-ID $ci received $pr request > $rm $ou source $si:$sp from $fu to $tu \n"); > xlog("L_INFO","=================================================== \n"); > > route(HANDLE_STIRSHAKEN); > > route(HANDLE_CANCEL); > > if (method == "INVITE"){ > route(RELAY); > } > } > > route[RELAY] { > # Sends a 300 Multiple Choices back to the proxy that requested the > routing lookup > xlog("L_INFO","RELAY - send reply \n"); > > file_out("cdr","RELAY|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d > %H:%M:%S)"); > send_reply("300", "Multiple Choices"); > exit; > } > > As for the HANDLE_STIRSHAKEN route, where the main process is: > > route[HANDLE_STIRSHAKEN] > { > > if (!is_method("INVITE")) { > return; > } > > $xavp(requestTime) = $utimef(%s); > $var(post) = $null; // resets and makes sure the $var(post) variable > is null before usage > > [.... get some header values ...] > > $var(res) = http_connect("api1", "/stsh", "application/json", > "$var(post)", "$var(result)"); > jansson_xdecode($var(result), "json"); > if ( $xavp(json=>sip-response-code) == 300 ) > { > remove_hf("Contact"); > append_to_reply("Contact: $xavp(json=>contact)\r\n"); > } > else > { > # add error message to reply > > sl_reply("$xavp(json=>sip-response-code)","$xavp(json=>sip-response-text)"); > exit; > } > } > > > > > Atenciosamente / Kind Regards / Cordialement / Un saludo, > > > *Sérgio Charrua* > > >
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected] To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender!
