Re: [Ntop] housekeeping - no enough memory

Thu, 24 Aug 2017 11:39:00 -0700 

Hi Roberto,
Can you monitor the ntopng memory usage over some hours to see if that is the process which consumes memory? 


I've just committed a fix for a memory leak on the Mac.cpp but it 
cannot affect you since you are receiving flows from nProbe. It will be 
very helpful if you could run ntopng with valgrind and send me the 
valgrind log (just the last 200 lines of log or so). Please execute

        sudo valgrind --leak-check=full ntopng /etc/ntopng.conf

Regards,
Emanuele


On Thu, Aug 24, 2017 at 6:40 PM, Roberto Alvarado 
<[email protected]> wrote:

Hi,


The problem is after some time, here is the log from the start of 
ntopng to the die of the process:



23/Aug/2017 11:20:05 [Ntop.cpp:1437] Setting local networks to 
[REMOVED]
23/Aug/2017 11:20:05 [Redis.cpp:109] Successfully connected to redis 
127.0.0.1:6379@0
23/Aug/2017 11:20:05 [NtopPro.cpp:183] [LICENSE] Reading license from 
/etc/ntopng.license
23/Aug/2017 11:20:05 [NtopPro.cpp:118] [LICENSE] /etc/ntopng.license: 
unable to validate Enterprise license [License mismatch error]
23/Aug/2017 11:20:05 [NtopPro.cpp:121] [LICENSE] /etc/ntopng.license: 
found valid Professional license
23/Aug/2017 11:20:06 [MySQLDB.cpp:680] Attempting to connect to MySQL 
for interface tcp://127.0.0.1:5556...
23/Aug/2017 11:20:06 [MySQLDB.cpp:720] Successfully connected to 
MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556
23/Aug/2017 11:20:06 [Ntop.cpp:1563] Registered interface 
tcp://127.0.0.1:5556 [id: 0]
23/Aug/2017 11:20:06 [main.cpp:252] PID stored in file 
/var/tmp/ntopng.pid
23/Aug/2017 11:20:06 [HTTPserver.cpp:797] Please read 
https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to 
enable SSL.

23/Aug/2017 11:20:06 [Utils.cpp:427] User changed to nobody

23/Aug/2017 11:20:06 [HTTPserver.cpp:851] Web server dirs 
[/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
23/Aug/2017 11:20:06 [HTTPserver.cpp:854] HTTP server listening on 
port(s) 3000
23/Aug/2017 11:20:06 [MySQLDB.cpp:667] Disconnected from MySQL for 
interface tcp://127.0.0.1:5556...
23/Aug/2017 11:20:06 [MySQLDB.cpp:680] Attempting to connect to MySQL 
for interface tcp://127.0.0.1:5556...
23/Aug/2017 11:20:06 [MySQLDB.cpp:720] Successfully connected to 
MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556

23/Aug/2017 11:20:06 [main.cpp:320] Working directory: /data/ntopng

23/Aug/2017 11:20:06 [main.cpp:322] Scripts/HTML pages directory: 
/usr/share/ntopng
23/Aug/2017 11:20:06 [Ntop.cpp:340] Welcome to ntopng x86_64 
v.3.1.170821 - (C) 1998-17 ntop.org
23/Aug/2017 11:20:06 [Ntop.cpp:345] Built on Debian GNU/Linux 8.2 
(jessie)
23/Aug/2017 11:20:06 [NtopPro.cpp:383] [LICENSE] System Id:     
[REMOVED]
23/Aug/2017 11:20:06 [NtopPro.cpp:384] [LICENSE] Edition:       
Professional
23/Aug/2017 11:20:06 [NtopPro.cpp:385] [LICENSE] License Type:  
Permanent License
23/Aug/2017 11:20:06 [NtopPro.cpp:388] [LICENSE] License:       
[REMOVED]
23/Aug/2017 11:20:06 [NtopPro.cpp:398] [LICENSE] Maintenance:   Until 
Thu Mar 22 12:28:01 2018 [211 days left]
23/Aug/2017 11:20:06 [PeriodicActivities.cpp:50] Started periodic 
activities loop...
23/Aug/2017 11:20:06 [NetworkInterface.cpp:2241] Started packet 
polling on interface tcp://127.0.0.1:5556 [id: 0]...
23/Aug/2017 11:20:07 [MySQLDB.cpp:680] Attempting to connect to MySQL 
for interface tcp://127.0.0.1:5556...
23/Aug/2017 11:20:07 [CollectorInterface.cpp:116] Collecting flows on 
tcp://127.0.0.1:5556
23/Aug/2017 11:20:08 [NetworkInterface.cpp:1202] WARNING: A flow has 
been seen from multiple exprters or from multiple IN/OUT interfaces. 
Check exporters configuration.
23/Aug/2017 12:07:26 [MySQLDB.cpp:720] Successfully connected to 
MySQL [localhost:ntop] for interface tcp://127.0.0.1:5556
23/Aug/2017 17:57:02 [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
23/Aug/2017 17:58:01 [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]



My config file:

root@mhost:/var/log/ntopng# cat /etc/ntopng/ntopng.conf
-G=/var/tmp/ntopng.pid
-i="tcp://127.0.0.1:5556"
-d="/data/ntopng"
-w="3000"
#-v=5
--max-num-hosts=300000
--local-networks=[REMOVED]
--dump-flows="mysql;localhost;ntop;[REMOVED]"
root@mhost:/var/log/ntopng#

A screenshoot of my dashboard:

https://i.dmtinc.cl/i/2017/08/24/CapturaNTOPNG428fd.png

More info from syslog:


Aug 22 22:20:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 22 22:21:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 22 22:22:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 22 22:23:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 22 22:24:00 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 22 22:24:09 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:10 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
memory]
Aug 22 22:24:10 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:11 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:12 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:13 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
memory]
Aug 22 22:24:13 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:14 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:15 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:16 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
memory]
Aug 22 22:24:16 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:17 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:18 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
Aug 22 22:24:19 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
memory]
Aug 22 22:24:19 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:20 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:21 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:22 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:22 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:23 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][/usr/share/ntopng/scripts/lua/modules/i18n/init.lua:186: 
not enough memory]
Aug 22 22:24:24 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:25 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][error loading 
module 'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:25 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:26 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:27 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:28 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][error loading 
module 'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]
Aug 22 22:24:28 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][error loading module 
'lua_utils' from file 
'/usr/share/ntopng/scripts/lua/modules/lua_utils.lua':#012#011not 
enough memory]

Aug 23 11:20:02 mhost logger: ntopng stop
Aug 23 11:20:05 mhost logger: ntopng start

Aug 23 11:20:08 mhost ntopng: [NetworkInterface.cpp:1202] WARNING: A 
flow has been seen from multiple exprters or from multiple IN/OUT 
interfaces. Check exporters configuration.
Aug 23 17:57:02 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory]
Aug 23 17:58:01 mhost ntopng: [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/minute.lua][not enough memory



Additional preferences:

{
  "ntopng.prefs.housekeeping_frequency": 5,
  "ntopng.prefs.local_host_cache_duration": 3600,
  "ntopng.prefs.local_host_max_idle": 300,
  "ntopng.prefs.non_local_host_max_idle": 60,
  "ntopng.prefs.flow_max_idle": 60,
  "ntopng.prefs.active_local_host_cache_interval": 3600,
  "ntopng.prefs.intf_rrd_raw_days": 1,
  "ntopng.prefs.intf_rrd_1min_days": 30,
  "ntopng.prefs.intf_rrd_1h_days": 100,
  "ntopng.prefs.intf_rrd_1d_days": 365,
  "ntopng.prefs.other_rrd_raw_days": 1,
  "ntopng.prefs.other_rrd_1min_days": 30,
  "ntopng.prefs.other_rrd_1h_days": 100,
  "ntopng.prefs.other_rrd_1d_days": 365,
  "ntopng.prefs.host_top_sites_creation": false,
  "ntopng.prefs.is_local_host_cache_enabled": true,
  "ntopng.prefs.is_active_local_host_cache_enabled": true,
  "ntopng.prefs.tiny_flows_export_enabled": true,
  "ntopng.prefs.max_num_alerts_per_entity": 1024,
  "ntopng.prefs.max_num_flow_alerts": 16384,
  "ntopng.prefs.flow_device_port_rrd_creation": false,
  "ntopng.prefs.disable_alerts_generation": true,
  "ntopng.prefs.probing_alerts": false,
  "ntopng.prefs.ssl_alerts": false,
  "ntopng.prefs.alerts_syslog": true,
  "ntopng.prefs.alerts.slack_notifications_enabled": false,
  "ntopng.prefs.alerts.dump_alerts_when_iface_is_alerted": false,
  "ntopng.prefs.max_num_packets_per_tiny_flow": 3,
  "ntopng.prefs.max_num_bytes_per_tiny_flow": 64,
  "ntopng.prefs.safe_search_dns": 2078163920,
  "ntopng.prefs.global_dns": 4294967295,
  "ntopng.prefs.secondary_dns": 4294967295,
  "ntopng.prefs.enable_captive_portal": false,
  "ntopng.prefs.redirection_url": "http:\/\/www.ntop.org",
  "ntopng.prefs.max_ui_strlen": 24,
  "ntopng.prefs.host_mask": 0,
  "ntopng.prefs.ldap.bind_dn": "",
  "ntopng.prefs.ldap.bind_pwd": "",
  "ntopng.prefs.ldap.search_path": "",
  "ntopng.prefs.ldap.user_group": "",
  "ntopng.prefs.ldap.admin_group": "",
  "ntopng.prefs.alerts.mysql_check_open_files_limit": "1",
  "ntopng.prefs.nagios_nsca_host": "localhost",
  "ntopng.prefs.nagios_nsca_port": "5667",

  "ntopng.prefs.nagios_send_nsca_executable": 
"\/usr\/local\/nagios\/bin\/send_nsca",
  "ntopng.prefs.nagios_send_nsca_config": 
"\/usr\/local\/nagios\/etc\/send_nsca.cfg",

  "ntopng.prefs.nagios_host_name": "ntopng-host",
  "ntopng.prefs.nagios_service_name": "NtopngAlert",
  "ntopng.prefs.default_snmp_community": "public",
  "ntopng.prefs.google_apis_browser_key": "",
  "ntopng.prefs.show_advanced_prefs": "true",
  "ntopng.prefs.host_blacklist": "0"
}

Regards!

Roberto

Sobre 23-08-2017 4:14:34, Emanuele Faranda <[email protected]> 
escribió:



_______________________________________________ Ntop mailing list 
[email protected] 
http://listgateway.unipi.it/mailman/listinfo/ntop

Hi Roberto,


What is the content of your ntopng.conf file? Does the problem occur 
immediately or after some time? Can you post a screenshot of your 
ntopng dashboard?


Emanuele


On Wed, Aug 23, 2017 at 1:46 AM, Roberto Alvarado 
<[email protected]> wrote:

Hi Folks,


My ntopng is crashing every time when the housekeeping is executed, 
the message of the log is explicit “no enough memory”, but this 
problems comes out after upgrade my install to the latest devel 
version, do you have any clue about this, of how much memory is 
needed?





21/Aug/2017 20:14:22 [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/second.lua][not enough memory]
21/Aug/2017 20:14:22 [Lua.cpp:6214] WARNING: Script failure 
[/usr/share/ntopng/scripts/callbacks/housekeeping.lua][not enough 
memory]


root@mhost:/var/log/ntopng# free -m

            total       used       free     shared    buffers     
cached
Mem:         32163      30833       1329        428        575      
25141

-/+ buffers/cache:       5116      27046
Swap:        18530        399      18131
root@mhost:/var/log/ntopng#

root@mhost:/var/log/ntopng# ps aux --sort -rss

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME 
COMMAND
nobody   26971 28.1  5.8 3375504 1942968 ?     Sl   16:12  76:01 
ntopng /etc/ntopng/ntopng.conf
mysql    19755 15.0  1.7 1809120 588236 ?      Ssl  Aug17 1173:54 
/usr/sbin/mysqld
root       838  0.0  0.1  90724 51236 ?        Ss   Feb17  14:29 
/lib/systemd/systemd-journald


root@mhost:/var/log/ntopng# ntopng -V
v.3.1.170821    [Enterprise/Professional build]

GIT rev:        
dev:d13a4a219d885e47a46b008c61fb5e24b49b4514:20170821

Pro rev:        r1130
Built on:       Debian GNU/Linux 8.2 (jessie)
System Id:
Edition:        Professional
License Type:   Permanent
Maintenance:    Until Thu Mar 22 12:28:01 2018 [211 days left]
License:
root@mhost:/var/log/ntopng#

Thanks

Regards
Roberto


_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop





















					Reply via email to