Are drops increasing on that socket while it is happening? ss -l src <local_interface> -u sport 5060 -m
On Tue, Aug 29, 2023 at 3:26 PM Ihor Olkhovskyi <[email protected]> wrote: > Just to add some info > > netstat -nlp > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State > PID/Program name > ... > udp 25167616 0 <local_interface>:5060 0.0.0.0:* > 211759/kamailio > ... > > So I see a huge Receive Queue on UDP for Kamailio which is not clearing. > > Le mar. 29 août 2023 à 14:29, Ihor Olkhovskyi <[email protected]> a > écrit : > >> Hello, >> >> I've faced a bit strange issue, but a bit of preface. I have Kamailio as >> a proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At some >> point presence server accepts around 5K PUBLISH within 1 minute and sending >> around the same amount of NOTIFY to proxy Kamailio. >> >> Proxy is "transforming" protocol to TLS, but at sime point I'm starting >> to get these type of errors >> >> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed >> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed >> <script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=tls> failed! >> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed >> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed >> >> Some of those messages are 100% valid as client can go away or so. Some >> are not, cause I'm sure client is alive and connected. >> >> But the problem comes later. At some moment proxy Kamailio just stops >> accept UDP traffic on this interface (where it also accepts all NOTIFY's), >> at the start of the "stopping accepting" Kamailio sends OPTIONS via >> DISPATCHER but not able to receive 200 OK. >> >> Over TLS on the same interface all is ok. On other (loopback) interface >> UDP is being processed fine, so I don't suspert some limit on open files >> here. >> >> Only restart of Kamailio proxy process helps in this case. >> >> I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in >> theory buffer should not be the case. >> >> Is there some internal "interface buffer" in Kamailio that is not freed >> upon failure send or maybe I've missed somethig? >> >> Kamailio 5.6.4 >> >> fork=yes >> children=12 >> tcp_children=12 >> >> enable_tls=yes >> >> tcp_accept_no_cl=yes >> tcp_max_connections=63536 >> tls_max_connections=63536 >> tcp_accept_aliases=no >> tcp_async=yes >> tcp_connect_timeout=10 >> tcp_conn_wq_max=63536 >> tcp_crlf_ping=yes >> tcp_delayed_ack=yes >> tcp_fd_cache=yes >> tcp_keepalive=yes >> tcp_keepcnt=3 >> tcp_keepidle=30 >> tcp_keepintvl=10 >> tcp_linger2=30 >> tcp_rd_buf_size=80000 >> tcp_send_timeout=10 >> tcp_wq_blk_size=2100 >> tcp_wq_max=10485760 >> open_files_limit=63536 >> >> Sysctl >> >> # To increase the amount of memory available for socket input/output >> queues >> net.ipv4.tcp_rmem = 4096 25165824 25165824 >> net.core.rmem_max = 25165824 >> net.core.rmem_default = 25165824 >> net.ipv4.tcp_wmem = 4096 65536 25165824 >> net.core.wmem_max = 25165824 >> net.core.wmem_default = 65536 >> net.core.optmem_max = 25165824 >> >> # To limit the maximum number of requests queued to a listen socket >> net.core.somaxconn = 128 >> >> # Tells TCP to instead make decisions that would prefer lower latency. >> net.ipv4.tcp_low_latency=1 >> >> # Optional (it will increase performance) >> net.core.netdev_max_backlog = 1000 >> net.ipv4.tcp_max_syn_backlog = 128 >> >> # Flush the routing table to make changes happen instantly. >> net.ipv4.route.flush=1 >> -- >> Best regards, >> Ihor (Igor) >> > > > -- > Best regards, > Ihor (Igor) > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to [email protected] > Important: keep the mailing list in the recipients, do not reply only to > the sender! > Edit mailing list options or unsubscribe: >
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
