Hi Henning, All good,I will re-examine the changes. There were positive changes after that. Before that , I had to restart kamailio around every week,
Quick question, is there any cleaner procedure to upgrade from 5.5.3 to 5.5.6. I have compiled from sources. Please understand these are production systems and should be done with 0 or minimal outage. Danish On Tue, Jun 6, 2023 at 10:14 PM Henning Westerholt <[email protected]> wrote: > > Hello Muhammad, > > maybe its related to the changes you've done, maybe its some load increase or > similar, but hard to say from the outside. > > As said, maybe try the minor update first, and then if its persists, lets go > the memory debugging route. > > Cheers, > > Henning > > -- > Henning Westerholt – https://skalatan.de/blog/ > Kamailio services – https://gilawa.com > > > -----Original Message----- > > From: Muhammad Danish Moosa <[email protected]> > > Sent: Dienstag, 6. Juni 2023 13:58 > > To: Henning Westerholt <[email protected]> > > Cc: Kamailio (SER) - Users Mailing List <[email protected]> > > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private memory > > from pkg > > pool > > > > Thanks for the response. I really appreciate it. > > > > This system was tested thoroughly in the lab but problems only appeared in > > production. Actually it was working smoothly for 3 months after I fixed some > > issues like external system calls etc and this appeared all of sudden. > > > > > > > > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <[email protected]> wrote: > > > > > > Hello, > > > > > > first step should be probably to update to the latest version of the > > > 5.5.x branch, > > e.g., 5.5.6. There have been several fixes, some also related to memory > > leaks > > included if I remember correctly. > > > > > > If the problem persists, it might be a new memory leak. This can be found > > > by > > enabling memory debugging, preferable on a test system or lab system. > > > https://www.kamailio.org/wiki/tutorials/troubleshooting/memory > > > > > > Cheers, > > > > > > Henning > > > > > > -- > > > Henning Westerholt – https://skalatan.de/blog/ Kamailio services – > > > https://gilawa.com > > > > > > > -----Original Message----- > > > > From: Muhammad Danish Moosa <[email protected]> > > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > > To: Kamailio (SER) - Users Mailing List > > > > <[email protected]> > > > > Subject: [SR-Users] Re: Kamailio - could not allocate private memory > > > > from pkg pool > > > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > > increasing very slowly but does not go back even after the calls are > > > > literally zero. I suspect some UDP listener process are not releasing > > > > the > > memory. > > > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > > variables or something , I want to understand how it works when > > > > calls are released. My understanding is usage should be minimum when > > > > calls are released. Looking for feedback please? > > > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > > real_used: 836144 > > > > real_used: 1017328 > > > > real_used: 1028656 > > > > real_used: 1020864 > > > > real_used: 1020632 > > > > real_used: 1006120 > > > > real_used: 1014536 > > > > real_used: 1023984 > > > > real_used: 1017584 > > > > real_used: 835784 > > > > real_used: 835784 > > > > real_used: 835784 > > > > real_used: 844344 > > > > real_used: 835784 > > > > real_used: 901344 > > > > real_used: 835784 > > > > real_used: 1259728 > > > > real_used: 1196688 > > > > real_used: 1181320 > > > > real_used: 1166624 > > > > real_used: 1216032 > > > > real_used: 1173464 > > > > real_used: 1168552 > > > > real_used: 1235528 > > > > real_used: 917904 > > > > > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > > restarting kamailio after every week or fortnight but I am looking > > > > for more stable solution. I am using kamailio 5.5.3 > > > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > > <[email protected]> wrote: > > > > > > > > > > Any recommendations? > > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > > [root@msteams-sip-01 ~]# free -m > > > > > total used free shared buff/cache > > > > > available > > > > > Mem: 7822 221 4055 386 3545 > > > > > 6900 > > > > > Swap: 5119 0 5119 > > > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > > <[email protected]> wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > I have been kamailio services working for few months and > > > > > > suddenly I got this error. > > > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable to > > > > > > set pv [$rp] > > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > > allocate private memory from pkg pool > > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot get > > > > > > pv spec for [$dlg_var([email protected])] > > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): could > > > > > > not allocate private memory from pkg pool > > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): out of > > > > > > mem > > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add > > > > > > branches > > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > > build_res_buf_from_sip_req(): could not allocate private memory > > > > > > from pkg pool > > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error > > > > > > reply > > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > > build_res_buf_from_sip_req(): could not allocate private memory > > > > > > from pkg pool > > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response building > > > > > > failed > > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > > delayed stateful reply failed > > > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > > > -- > > > > > > Muhammad Danish Moosa > > > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > > Christopher McCandless > > > > > > > > > > > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > > > > > > > > > > > > -- > > > > Muhammad Danish Moosa > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > Christopher McCandless > > > > __________________________________________________________ > > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > > unsubscribe send an email to [email protected] > > > > Important: keep the mailing list in the recipients, do not reply only > > > > to the > > sender! > > > > Edit mailing list options or unsubscribe: > > > > > > > > -- > > Muhammad Danish Moosa > > > > " The core of mans' spirit comes from new experiences. "___ Christopher > > McCandless -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
