Bastian,
According to mine stats all seems ok...
# kamctl stats shmem
{
"jsonrpc": "2.0",
"result": [
"shmem:fragments = 330",
"shmem:free_size = 1889418024",
"shmem:max_used_size = 258962624",
"shmem:real_used_size = 258065624",
"shmem:total_size = 2147483648",
"shmem:used_size = 248809096"
],
"id": 212042
}
pkg.stats also not showing any signs of exhausting.
For me what is working is server can send packets, but not receive em
back even. And it's happening only on 1 interface.
Le 30/08/2023 à 21:44, Bastian Triller a écrit :
I'm asking, because we kind of observed similar behaviour after
overload. We've did some load testing and at some point SHM was
exhausted and that didn't recover itself. One process was using nearly
100% CPU. And I guess that at least sl replies still works at that point.
On Wed, Aug 30, 2023 at 9:28 PM Ihor Olkhovskyi
<[email protected]> wrote:
Bastian,
I'm fine with retransmissions and so, but for me it's really
interesting, that server is not restoring it's state after a while
and only restart helps. And this is happening only with UDP on one
interface, others are working fine over UDP
Le 30/08/2023 à 17:19, Bastian Triller a écrit :
Maybe it's not beneficial to have a receive buffer that high. If
your presence server is stateful, it will send retransmissions if
packets get dropped. But it will also send retransmissions if
your proxy does not reply fast enough. So your proxy may be
damped by retransmissions. Did you check SHM/active transactions
on your proxy?
Regards,
Bastian
On Wed, Aug 30, 2023 at 3:56 PM Ihor Olkhovskyi
<[email protected]> wrote:
They are increasing, actually
# ss -l -u -m src X.X.X.X/Y
State Recv-Q Send-Q Local
Address:Port Peer Address:Port
UNCONN 25167616 0 X.X.X.X:sip *:*
skmem:(r25167616,rb25165824,t0,tb65536,f2304,w0,o0,bl0,d514894)
Le mer. 30 août 2023 à 15:04, Bastian Triller
<[email protected]> a écrit :
Are drops increasing on that socket while it is happening?
ss -l src <local_interface> -u sport 5060 -m
On Tue, Aug 29, 2023 at 3:26 PM Ihor Olkhovskyi
<[email protected]> wrote:
Just to add some info
netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
...
udp 25167616 0 <local_interface>:5060
0.0.0.0:* 211759/kamailio
...
So I see a huge Receive Queue on UDP for Kamailio
which is not clearing.
Le mar. 29 août 2023 à 14:29, Ihor Olkhovskyi
<[email protected]> a écrit :
Hello,
I've faced a bit strange issue, but a bit of
preface. I have Kamailio as a proxy (TLS/WS <->
UDP) and second Kamailio as a presence server. At
some point presence server accepts around 5K
PUBLISH within 1 minute and sending around the
same amount of NOTIFY to proxy Kamailio.
Proxy is "transforming" protocol to TLS, but at
sime point I'm starting to get these type of errors
tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
tm [t_fwd.c:1588]: t_send_branch(): sending
request on branch 0 failed
<script>: [RELAY] Relay to
<sip:X.X.X.X:51571;transport=tls> failed!
tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
tm [t_fwd.c:1588]: t_send_branch(): sending
request on branch 0 failed
Some of those messages are 100% valid as client
can go away or so. Some are not, cause I'm sure
client is alive and connected.
But the problem comes later. At some moment proxy
Kamailio just stops accept UDP traffic on this
interface (where it also accepts all NOTIFY's),
at the start of the "stopping accepting" Kamailio
sends OPTIONS via DISPATCHER but not able to
receive 200 OK.
Over TLS on the same interface all is ok. On
other (loopback) interface UDP is being processed
fine, so I don't suspert some limit on open files
here.
Only restart of Kamailio proxy process helps in
this case.
I've tuned net.core.rmem_max and
net.core.rmem_default to 25 Mb, so in theory
buffer should not be the case.
Is there some internal "interface buffer" in
Kamailio that is not freed upon failure send or
maybe I've missed somethig?
Kamailio 5.6.4
fork=yes
children=12
tcp_children=12
enable_tls=yes
tcp_accept_no_cl=yes
tcp_max_connections=63536
tls_max_connections=63536
tcp_accept_aliases=no
tcp_async=yes
tcp_connect_timeout=10
tcp_conn_wq_max=63536
tcp_crlf_ping=yes
tcp_delayed_ack=yes
tcp_fd_cache=yes
tcp_keepalive=yes
tcp_keepcnt=3
tcp_keepidle=30
tcp_keepintvl=10
tcp_linger2=30
tcp_rd_buf_size=80000
tcp_send_timeout=10
tcp_wq_blk_size=2100
tcp_wq_max=10485760
open_files_limit=63536
Sysctl
# To increase the amount of memory available for
socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824
# To limit the maximum number of requests queued
to a listen socket
net.core.somaxconn = 128
# Tells TCP to instead make decisions that would
prefer lower latency.
net.ipv4.tcp_low_latency=1
# Optional (it will increase performance)
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 128
# Flush the routing table to make changes happen
instantly.
net.ipv4.route.flush=1
--
Best regards,
Ihor (Igor)
--
Best regards,
Ihor (Igor)
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial
Discussions
To unsubscribe send an email to
[email protected]
Important: keep the mailing list in the recipients,
do not reply only to the sender!
Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to
[email protected]
Important: keep the mailing list in the recipients, do
not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Best regards,
Ihor (Igor)
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not
reply only to the sender!
Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email [email protected]
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the
sender!
Edit mailing list options or unsubscribe:
