hishamway left a comment (kamailio/kamailio#4808)
Hi, It is observed from our production servers, we are heavily dependent on
event_route[nats:presence] {
xlog("L_INFO", "received event payload [$natsData]");
pua_json_publish($natsData);
}
to update presence info from our media server.
We are seeing a 2% total memory spike for one of the child process in kamailio,
up on inspection it the child process handling our nat's subscriber.
We are seeing this tread in our multiple production instance, currently we are
restarting the server once it reaches 80%, As this is seen for some time now no
sudden spike in memory is not seen till date.
I will attach a new memory report from one of our product server which just hit
80% of total memory.
So checked with shm memory stats and pkg mem stats , there is no significant
increase in that.
##
```
admin@sipcore2:~$ ps aux | grep kamailio
admin 822540 0.0 0.0 3880 1968 pts/0 S+ 11:51 0:00 grep kamailio
kamailio 3681511 0.0 0.9 377520 37164 ? SNl Jun05 0:00
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681515 0.2 2.6 378092 103100 ? SN Jun05 130:03
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681516 0.2 2.5 378140 102632 ? SN Jun05 128:53
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681517 0.2 2.6 378084 103784 ? SN Jun05 129:42
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681518 0.2 2.5 378092 102608 ? SN Jun05 128:48
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681519 0.2 2.6 378128 103096 ? SN Jun05 129:21
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681520 0.2 2.5 378076 102876 ? SN Jun05 129:28
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681521 0.2 2.5 378072 102752 ? SN Jun05 128:03
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681522 0.2 2.6 378204 103684 ? SN Jun05 128:47
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681523 0.2 2.2 377664 87584 ? SN Jun05 101:25
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681524 0.2 2.2 377664 87268 ? SN Jun05 101:38
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681525 0.2 2.1 377664 86784 ? SN Jun05 100:42
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681526 0.2 2.1 377664 86432 ? SN Jun05 101:02
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681527 0.0 2.1 377664 83216 ? SN Jun05 1:32
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681528 0.0 2.1 377664 85312 ? SN Jun05 45:40
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681529 0.1 2.0 378116 82140 ? SN Jun05 79:22
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681530 0.1 2.1 377664 86696 ? SN Jun05 77:25
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681531 0.0 2.0 377664 81360 ? SN Jun05 1:22
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681532 0.0 2.0 377664 80540 ? SN Jun05 0:18
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681533 0.0 1.9 377664 78320 ? SN Jun05 0:04
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681534 0.0 1.9 377664 78364 ? SN Jun05 0:19
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681535 0.0 1.9 377664 79156 ? SN Jun05 0:20
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681536 0.0 1.9 377664 78096 ? SN Jun05 0:16
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681537 0.0 1.9 377664 78480 ? SN Jun05 0:17
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681538 0.0 1.7 377664 69244 ? SN Jun05 0:03
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681539 0.0 2.1 379968 83180 ? SN Jun05 17:05
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681540 0.0 0.8 377664 33344 ? SN Jun05 2:13
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681541 0.0 2.0 377664 79832 ? SN Jun05 16:21
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681542 0.0 2.0 377664 82904 ? SN Jun05 16:19
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681543 0.0 0.2 443200 11704 ? SNl Jun05 0:02
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681544 0.0 62.8 2868032 2489372 ? SNl Jun05 28:04
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681545 0.0 1.9 443244 75500 ? SNl Jun05 2:53
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681546 0.0 1.8 443244 73480 ? SNl Jun05 3:31
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681547 0.0 0.3 377664 12420 ? SN Jun05 0:00
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681551 0.0 0.3 377664 15400 ? SN Jun05 2:00
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681559 0.0 0.3 377664 13304 ? SN Jun05 0:37
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681560 0.0 0.3 377664 13304 ? SN Jun05 0:37
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681561 0.0 0.3 377664 13304 ? SN Jun05 0:37
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
kamailio 3681562 0.0 0.4 377520 16628 ? SN Jun05 0:29
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
```
As you can see one specific child process is taking too much memory
```
kamailio 3681544 0.0 62.8 2868032 2489372 ? SNl Jun05 28:04
/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/main.cfg -m
256 -M 32 --atexit=no
```
3681544
NATS Subscriber
That's process is NATS subscriber
This is happening repeatedly, but gradual increase of 2% memory increase on a 4
GB Ram machine.
Attaching the memory analysis from grafana of last 30 days
<img width="1517" height="328" alt="Image"
src="https://github.com/user-attachments/assets/97b8f4b1-795c-42a0-88fe-be1ecc7100bd";
/>
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4808#issuecomment-4925272122
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4808/[email protected]>_______________________________________________
Kamailio - Development Mailing List -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the
sender!
