I'll refer you to the usual answers in the list and the docs/FAQ... regarding hash sizing.
However, there were some deadlock issues in and around .52 or .53 (I mis-remember) and some other issues with the locking. You should be running .54 - source @ SourceForge. Random thoughts... How many hosts are you monitoring? With a /20 and a /24, you're an ISP with 17K hosts. If each of your hosts actively talks to a few others - you'll have host records for everything referred to on both sides of every single session - so 70K isn't outlandish. Add in a couple of port scans and pow! Yes, the hash purge will EVENTUALLY purge them, but for performance reasons it won't happen quickly. Look into the dynamic hash purge parameter (whatever it's called). You may have to turn on some of the large system / less data switches - look in the FAQ for --border-sniffer-mode, the new (2.1.5x) switches are documented there also. Since you're running a recent snapshot, the info on the textinfo.html page should show you the hash sizes, etc. But the first recommendation is to use .54 -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Ashley Gates Sent: Monday, January 20, 2003 3:44 PM To: [EMAIL PROTECTED] Subject: [Ntop] Hash size / memory gobbler I've popped this question before, but semi-inconclusive evidence and info on the problem. I'll start from the top, reader's digest version: Hardware: PII-450 1gig RAM 20 gig drive Software: Redhat 7.3 ntop v.2.1.53 MT (SSL) [i686-pc-linux-gnu] (01/13/03 02:19:38 AM build) NetFlow Router: Cisco 7204 VXR passing 20-25 mbits/sec via FastEthernet to upstream provider. The main purpose for my ntop experience is to use the Ntop for NetFlow collection and processing of NFC data. My CLI looks like this: ntop -P /usr/local/share/ntop -u ntop -M -n -p /usr/local/share/ntop/protocols.list -W 3001 -w 0 -C -m xxx.xxx.xxx.xxx/20,xxx.xxx.xxx.xxx/24 The moment I start Ntop the hash size 'extends' 512,1024,etc.. While watching top I see memory consumption grow rapidly. After 30 minutes I will see ntop eat 875 megs of RAM! CPU stays healthy with 25-60 percent load at times while ntop is running. Shortly after it will fail with the following screen shot: Wait please: ntop is coming up... 20/Jan/2003 14:36:13 Initializing IP services... 20/Jan/2003 14:36:13 Initializing SSL... 20/Jan/2003 14:36:13 SSL initialized successfully 20/Jan/2003 14:36:13 Initializing GDBM... 20/Jan/2003 14:36:13 Initializing network devices... 20/Jan/2003 14:36:13 ntop v.2.1.53 MT (SSL) [i686-pc-linux-gnu] (01/13/03 02:19:38 AM build) 20/Jan/2003 14:36:13 Listening on [eth0] 20/Jan/2003 14:36:13 Copyright 1998-2002 by Luca Deri <[EMAIL PROTECTED]> 20/Jan/2003 14:36:13 Get the freshest ntop from http://www.ntop.org/ 20/Jan/2003 14:36:13 Initializing... 20/Jan/2003 14:36:13 Truncated network size to 1024 hosts (real netmask 255.255.255.224) 20/Jan/2003 14:36:13 Loading plugins (if any)... 20/Jan/2003 14:36:13 Searching plugins in /usr/local/lib/ntop/plugins 20/Jan/2003 14:36:13 Welcome to icmpWatchPlugin. (C) 1999 by Luca Deri. 20/Jan/2003 14:36:13 Welcome to LastSeenWatchPlugin. (C) 1999 by Andrea Marangoni. 20/Jan/2003 14:36:13 Welcome to NetFlow. (C) 2002 by Luca Deri. 20/Jan/2003 14:36:13 Welcome to nfsWatchPlugin. (C) 1999 by Luca Deri. 20/Jan/2003 14:36:13 Welcome to PDAPlugin. (C) 2001-2002 by L.Deri and W.Brock 20/Jan/2003 14:36:13 Welcome to sFlowPlugin. (C) 2002 by Luca Deri. 20/Jan/2003 14:36:13 RRD: Welcome to rrdPlugin. (C) 2002 by Luca Deri. 20/Jan/2003 14:36:13 Device 0(eth0) DLT_ is 1, assuming mtu 1514, header 14 20/Jan/2003 14:36:13 Processing protocol file: '/usr/local/share/ntop/protocols.list', size: 370 20/Jan/2003 14:36:13 Resetting traffic statistics... 20/Jan/2003 14:36:14 Started thread (1026) for network packet analyser. 20/Jan/2003 14:36:14 Started thread (2051) for idle hosts detection. 20/Jan/2003 14:36:14 Initializing plugins (if any)... 20/Jan/2003 14:36:14 Starting plugin 'rrdPlugin'... 20/Jan/2003 14:36:14 Starting plugin 'sFlowPlugin'... 20/Jan/2003 14:36:14 Starting plugin 'PDAPlugin'... 20/Jan/2003 14:36:14 Starting plugin 'nfsWatchPlugin'... 20/Jan/2003 14:36:14 Starting plugin 'NetFlow'... 20/Jan/2003 14:36:14 NetFlow collector listening on port 2055. 20/Jan/2003 14:36:14 NetFlow export disabled 20/Jan/2003 14:36:14 Starting plugin 'LastSeenWatchPlugin'... 20/Jan/2003 14:36:14 Starting plugin 'icmpWatchPlugin'... 20/Jan/2003 14:36:14 Waiting for HTTPS (SSL) connections on port 3001... 20/Jan/2003 14:36:14 Started thread (4101) for web server. 20/Jan/2003 14:36:14 Sniffying... 20/Jan/2003 14:36:14 Started thread (5126) for network packet sniffing on eth0. 20/Jan/2003 14:36:14 Welcome to NetFlow: listening on UDP port 2055... 20/Jan/2003 14:36:14 Started thread (3076) for NetFlow. 20/Jan/2003 14:36:14 Extending hash size [newSize=512][deviceId=1] 20/Jan/2003 14:36:16 Extending hash size [newSize=1024][deviceId=1] 20/Jan/2003 14:36:20 Extending hash size [newSize=2048][deviceId=1] 20/Jan/2003 14:36:28 Extending hash size [newSize=4096][deviceId=1] 20/Jan/2003 14:36:47 Extending hash size [newSize=8192][deviceId=1] 20/Jan/2003 14:37:14 IDLE_PURGE: purgeIdleHosts firstRun (mutex every 64 times through the loop) 20/Jan/2003 14:37:36 Extending hash size [newSize=12288][deviceId=1] 20/Jan/2003 14:38:41 Extending hash size [newSize=16384][deviceId=1] 20/Jan/2003 14:39:55 Extending hash size [newSize=20480][deviceId=1] 20/Jan/2003 14:41:21 Extending hash size [newSize=24576][deviceId=1] 20/Jan/2003 14:43:32 Extending hash size [newSize=28672][deviceId=1] 20/Jan/2003 14:45:49 Extending hash size [newSize=32768][deviceId=1] 20/Jan/2003 14:46:57 Extending hash size [newSize=36864][deviceId=1] 20/Jan/2003 14:49:08 Extending hash size [newSize=40960][deviceId=1] 20/Jan/2003 14:50:46 Extending hash size [newSize=45056][deviceId=1] 20/Jan/2003 14:53:46 Extending hash size [newSize=49152][deviceId=1] 20/Jan/2003 14:56:02 Extending hash size [newSize=53248][deviceId=1] 20/Jan/2003 14:58:54 Extending hash size [newSize=57344][deviceId=1] 20/Jan/2003 15:01:45 Extending hash size [newSize=61440][deviceId=1] 20/Jan/2003 15:05:04 Extending hash size [newSize=65536][deviceId=1] 20/Jan/2003 15:08:10 Extending hash size [newSize=69632][deviceId=1] 20/Jan/2003 15:39:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 15:46:54 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 15:48:37 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 15:55:49 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 15:59:06 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:05:05 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:10:01 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:18:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:21:20 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:21:26 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:21:28 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:22:07 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:22:58 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:23:07 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:23:15 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:23:25 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:24:17 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:24:55 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:25:30 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:25:54 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:26:01 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:27:46 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:28:05 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:28:12 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:28:17 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:28:44 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:29:03 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:29:16 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:29:25 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:30:11 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:30:22 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:30:51 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:31:29 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:31:39 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:31:57 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:31:59 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:32:39 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:20 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:23 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:27 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:33:59 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:20 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:20 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:32 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:41 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:54 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:34:54 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:08 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:17 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:22 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:26 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:41 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:35:51 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:03 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:15 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:28 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:43 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:48 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:36:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:37:07 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:37:08 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:37:30 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:37:37 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:37:56 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:17 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:25 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:27 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:28 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:29 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:30 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:35 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:36 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:38 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:41 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:42 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:53 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:38:55 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] 20/Jan/2003 16:39:18 WARNING: accessMutex() call with a self-LOCKED mutex [from pbuf.c:519 updateInterfacePorts, locked by updateInterfacePorts] /usr/local/bin/startntop: line 1: 29126 Killed ANY IDEAS? Thanks, Ashley Gates Director of Network Operations Atlanta Broadband, Inc. http://www.atlantabroadband.net Office: 770.794.0767 _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop
