Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
@miconda please look into this one. because now a days developer already started to move for kamailio in lua. if there is memory leak then it will be great issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-412337043___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
Thank you for the updates. :-) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-408659508___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
Didn't get the chance to look at it, being in a trip to USA (Cluecon). Hope to get back to it soon, after returning. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-408409299___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
@miconda please let me know if this issue is reproduced by the same config at your side or not -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-408352787___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
@miconda are you able to reproduce it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-406842466___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
yes i am using master branch only. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-404770193___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
So, I assume it is still there in the master branch, after my last commit realted to app_lua. I will try to reproduce here with the first available time frame. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-404753743___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
this can be reporduced by following lua config
kamailio.cfg
```
#!KAMAILIO
### Defined Values #
#!substdef "!MY_IP_ADDR!!g"
#!substdef "!MY_EXTERNAL_IP!!g"
#!substdef "!MY_UDP_PORT!5060!g"
#!substdef "!MY_UDP_ADDR!udp:MY_IP_ADDR:MY_UDP_PORT!g"
# Dispatcher File
#!define DISPATCHER_FILE "/etc/kamailio/dispatcher.list"
# Routing File
#!define ROUTE_LUA "/etc/kamailio/route.lua"
### Global Parameters #
## LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR
#!ifdef WITH_DEBUG
debug = 4
log_stderror = yes
#!else
debug = 2
log_stderror = no
#!endif
memdbg = 5
memlog = 5
# log setting
log_facility = LOG_LOCAL0
listen = MY_UDP_ADDR advertise MY_EXTERNAL_IP:MY_UDP_PORT
children = 8
async_workers=8
### Modules Section
#!ifdef WITH_SRCPATH
mpath = "modules/"
#!else
mpath = "/usr/local/lib64/kamailio/modules/"
#!endif
# -- module loading --
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "maxfwd.so"
loadmodule "nathelper.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "path.so"
loadmodule "jsonrpcs.so"
loadmodule "app_lua.so"
loadmodule "dispatcher.so"
loadmodule "json.so"
loadmodule "cfg_rpc.so"
loadmodule "ctl.so"
# - setting module-specific parameters ---
# - rr params -
# set next param to 1 to add value to ;lr param (helps with some UAs)
modparam("rr", "enable_full_lr", 0)
# do not append from tag to the RR (no need for this script)
modparam("rr", "append_fromtag", 1)
# - dispatcher params -
modparam("dispatcher", "list_file", DISPATCHER_FILE)
modparam("dispatcher", "ds_probing_mode", 1)
modparam("dispatcher", "flags", 2)
modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_ping_from", "sip:p...@test.my.com")
# - nathelper params
modparam("nathelper", "received_avp", "$avp(s:rcv)")
# - ctl --
modparam("ctl", "binrpc", "tcp:MY_IP_ADDR:2046")
# - app_lua params
modparam("app_lua", "reload", 1)
modparam("app_lua", "load", ROUTE_LUA)
# - tm params -
# auto-discard branches from previous serial forking leg
modparam("tm", "failure_reply_mode", 3)
modparam("tm", "fr_timer", 5000)
modparam("tm", "fr_inv_timer", 12)
modparam("tm", "restart_fr_on_each_reply", 1)
modparam("tm", "pass_provisional_replies", 1)
modparam("tm", "contacts_avp", "tm_contacts");
modparam("tm", "contact_flows_avp", "tm_contact_flows");
cfgengine "lua"
route.lua
```
function ksr_request_route()
local request_method = headers.get("$rm") or "";
local user_agent = headers.get("$ua") or "";
KSR.info("recieve request " .. request_method);
KSR.exit();
return 1;
end
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1577#issuecomment-404720655___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
Updates: I am running test with master branch. i will report if i found memory leak with simple config so you can reproduce. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-403902230___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
An update -- I pushed a commit to restore the Lua execution stack. Try with latest master and if the issue is still expose, try to give the minimal configs and scripts to reproduce, as I wrote in my previous comment. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-403480407___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
Did you run and test it long enough that the memory increased a lot? I checked the logs for couple of PIDs and no relevant leak is reported. Maybe you can make minimal kamailio.cfg and kemi lua script along with a sipp scenario that I can use and reproduce here. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-403461159___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
[vg.log](https://github.com/kamailio/kamailio/files/2173037/vg.log) Please check the valgrind log. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-403238200___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
I do not see any new relevant information attached here, have you run with valgrind as I metioned in my previous comment? It should give the report if there are leaks for system memory. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-402988071___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
@miconda please see this issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-402530641___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
yeah sure i will run that one. i am using both ways 1. using lua modules 2. without using lua modules both ways i can figure out that system memory getting increases. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401816921___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
Are you loading and using any Lua modules/libraries? Or it is just using the KSR module exported by Kamailio? Are you issuing often reloads of lua script? The way to troubleshoot leak of system memory is to run kamailio via valgrind (http://valgrind.org/). Do not forget you have to track forked processes: ``` valgrind --trace-children=yes ... ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401814245___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
we are seeing the memory related system increasing to 97% so its might the case that system memory getting increase. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401811824___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
It is not clear what type of memory leaks? Is it the on managed by kamailio in pkg or shm? Or is the system memory? Because in the shell script you use system's `free`. To watch kamailio's shm memory do: ``` kamctl stats shmem ``` For pkg: ``` kamctl rpc pkg.stats ``` If it is system memory, then the logs of kamailio related to qm_status are irrelevant. So once it is clarified what memory type is leaking, then we can provide next steps to analyze the issue. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401809431___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
more log for pkg_memory ``` 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_load_script(136) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_reload_script(451) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(116) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(122) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/dset.c: init_dst_set(83) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(121) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(91) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(93) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/nonsip_hooks.c: init_nonsip_hooks(43) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_init_buffer(2045) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/route.c: route_new_list(200) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/rpc_lookup.c: rpc_hash_add(146) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_hostname(1301) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_socket_list(1509) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(420) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(437) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(230) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(235) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(252) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(264) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(273) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/sr_module.c: init_modules(1026) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/timer_proc.c: sr_wtimer_init(308) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/udp_server.c: udp_rcv_loop(436) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_fifo_mod_init(525) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_init_fifo_server(123) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_sock.c: jsonrpc_dgram_mod_init(182) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from pv: pv_shv.c: pv_get_shvar(362) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from xlog: xlog.c: mod_init(214) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: dumping all alloc'ed. fragments: 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_load_script(136) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_reload_script(451) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(116) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(122) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/dset.c: init_dst_set(83) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(121) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(91) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(93) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core:
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
script to memory count for kamailio process ``` #!/bin/bash free echo "" for line in $(pgrep -f '/usr/local/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio.pid -m 16 -M 8 -u root -g root') do pms=$(pmap -x $line|grep total) echo "$pms pid:$line" done ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401611314___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [kamailio/kamailio] MemoryUtilization getting increase with lua routing (#1577)
@miconda rightnow our memory utilization are 97%. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1577#issuecomment-401450253___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
