Bogdan,
on the augmenting drop_requests,
drop is not used in that script. Neither is forward. Everything is
t_relay. There is no b2b.
The thing is that we observed drops on udp level.
We followed the recommendations of Ovidiu Sas's presentation of last
year in Houston:
- increase PKG mem
- increase SHM mem
- increase workers to 24 so that the queue empties faster.
- we checked the udp queues on linux level and we saw drops there.
=> we augmented them to 50 megs (sysctl -w
net.core.rmem_max=52428800 and sysctl -w net.core.rmem_default=52428800)
and the drops on OS level where gone.
Also worker and memory load are max 30 %.
Hence we thought that we were okay, but still drops on opensips level.
Net result was that this node in the system lost all connection with the
destination of the loadbalancer although it received keep alive options
responses from the loadbalancer destination on its NIC (we could see
that in a continuously running tcpdump).
=> hence it seems that it is opensips's receive buffer that is too small
(as I read the description : "Returns the number of requests dropped
even before entering the script routing logic.", I thought that this
pointed to the receive buffer of opensips). All of this is happening on
a physical machine on which two other opensips instances are running
also. Interestingly enough the problem is only observed in the instance
that handles registrations and invites (1600 REG/ s and 300 INV /s).
Therefore we dived a bit deeper and came on this MAX_RECV_BUFFER_SIZE
262144 (which is the default udp queue size setting on linux). Could
this be related somehow ?
Secondly, what would the recommendation be for scaling a system like this ?
On 18/04/2024 16:29, Bogdan-Andrei Iancu wrote:
The `drop_requests` statistic is incremented when:
* the request is dropped by a pre-script callback (like B2B when there
is no script execution for certain messages)
* the stateless `forward()` core function failed to send out something.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 18.04.2024 17:19, Johan De Clercq wrote:
No I don't.
what I find strange is that MAX_RECV_BUFFER_SIZE 262144 is the
default value of net.core.rmem_max and net.core.rmem_default.
Op do 18 apr 2024 om 16:02 schreef Ben Newlin <ben.new...@genesys.com>:
Are you calling drop() anywhere in your script?
https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13
Ben Newlin
*From: *Users <users-boun...@lists.opensips.org> on behalf of
Johan De Clercq <jo...@democon.be>
*Date: *Thursday, April 18, 2024 at 5:27 AM
*To: *OpenSIPS users mailling list <users@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Users] question on core statistics.
* EXTERNAL EMAIL - Please use caution with links and attachments *
------------------------------------------------------------------------
would it make sense to recompile with other flags ? And how do I
set them (I don't find these of menuconfig's compile options)?
Currently it has MAX_RECV_BUFFER_SIZE 262144 and BUF_SIZE 65535.
Can somebody explain also what both flags mean.
Op do 18 apr 2024 om 11:07 schreef Johan De Clercq
<jo...@democon.be>:
would it make sense to recompile with other flags ?
Currently it has MAX_RECV_BUFFER_SIZE 262144 and BUF_SIZE 65535.
Can somebody explain also what both flags mean.
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
Op do 18 apr 2024 om 10:32 schreef Johan De Clercq
<jo...@democon.be>:
Guys,
I have an opensips instance running with 24 worker children.
The worker load is very low.
UDP queues are on 50 megs.
when i query via the OS
cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr
tm->when retrnsmt uid timeout inode ref pointer drops
590: 03231D0A:13C4 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 413684019 2
ffff880074820bc0 0
591: 03231D0A:13C5 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 413766438 2
ffff880465e4a440 0
592: 03231D0A:13C6 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 412035865 2
ffff8803e5a56b80 0
934: 01231D0A:151C 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26790 2
ffff88046c054840 0
935: 0201FFEF:151D 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26787 2
ffff88046c054bc0 0
935: 01231D0A:151D 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26791 2
ffff88046c0544c0 0
1972: 00000000:D92A 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 15506 2
ffff88046dce5040 0
5479: 00000000:E6DD 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 22811 2
ffff880465e4ab40 0
12075: AA0914AC:00A1 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 20572 2
ffff88086d020800 0
12075: 0100007F:00A1 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 20571 2
ffff88086d020b80 0
13320: 00000000:857E 00000000:0000 07 00000000:00000000
00:00000000 00000000 100 0 17515 2
ffff8800368ac780 0
15661: 00000000:CEA3 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 15505 2
ffff8800368acb00 0
=> no drops
what worries me is that there are drop requests and they
go up when I query via the mi interface
opensipsctl fifo get_statistics drop_requests
core:drop_requests:: 198107
opensipsctl fifo get_statistics drop_requests
core:drop_requests:: 199157
opensipsctl_reg fifo get_statistics drop_requests
core:drop_requests:: 204116
I don't see any memory issue, also the processload is low.
so 3 questions:
- what exactly is drop_request.
- do I need to worry about this
- how can I make them go lower.
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
