Rajendra Kishore Bonumahanti created TS-4266:
------------------------------------------------
Summary: ATS memory statistics shows that memory utilization is
doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find
enough memory.
Key: TS-4266
URL: https://issues.apache.org/jira/browse/TS-4266
Project: Traffic Server
Issue Type: Bug
Components: Lua
Reporter: Rajendra Kishore Bonumahanti
ATS memory statistics shows memory utilization is doubled after “traffic_ctl
config reload”. We get “not enough memory” error in the subsequent attempt and
“config reload” fails.
ATS is configured with 100 map entries in remap.config, all share the same lua
script.
ATS is started: The memory information is..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 1416092 670256 663736
After 1st Config reload:
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 1932660 1128084 1121544
After 2nd config reload: It had failed with error “not enough memory” and
memory status as..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 2170756 1167808 1160836
Error displayed in diags.log:
=======================
[Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: Failed to create new
instance for plugin /opt/trafficserver/libexec/trafficserver/tslua.so (not a
TS_SUCCESS return)
[Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: Could not add rule at
line #3; Aborting!
[Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: [ReverseProxy] Can't
create new remap instance for plugin
"/opt/trafficserver/libexec/trafficserver/tslua.so" - [ts_lua_add_module]
luaL_loadfile /opt/trafficserver/etc/trafficserver/lua/process_remap.lua
failed: not enough memory at line 3
[Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: something failed during
BuildTable() -- check your remap plugins!
[Mar 8 23:27:27.595] Server {0x2af92498b700} WARNING: failed to reload
remap.config, not replacing!
Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory:
'..collectgarbage("count"))
[Mar 8 23:27:27.579] Server {0x2af92498b700} DIAG: (ts_lua) __init__(): Lua VM
memory: 3629.7060546875
This shows that Lua VMs are hitting the max capacity.
Solution:
=======
I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and
ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for
the instance. However the lua memory is not released and reused.
So, I have added code to start the garbage collector in ts_lua_del_module() .
int
ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n)
{
….
lua_newtable(L);
lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY */
lua_gc(L, LUA_GCCOLLECT, 0);
TSMutexUnlock(arr[i].mutexp);
}
return 0;
}
This has improved the situation. However, I also added garbage collection in
ts_lua_add_module() at the end. With these two additions, we have tested the
code, the memory utilization is stable and we could do config reload at lest
100 times with the background load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
