(repost after reject)
netFlowPlugin.c calls checkSessionIdx which is this (and that's what is
reporting the error):
u_int _checkSessionIdx(u_int idx, int actualDeviceId, char* file, int line)
{
if(idx > myGlobals.device[actualDeviceId].actualHashSize) {
traceEvent(TRACE_ERROR, "Index error idx=%u/deviceId=%d:0-%d @
[%s:%d]\n",
idx, actualDeviceId,
myGlobals.device[actualDeviceId].actualHashSize,
file, line);
return(0); /* Last resort */
} else
return(idx);
}
32768 used to be a hard limit on the hash size (why I'm not really clear on,
but it was):
ntop.h: 1313 #define MAX_HASH_SIZE 32768
There are now NO references to MAX_HASH_SIZE, so it should be able to grow
larger. However, each hash entry takes up a LOT of memory. So having
ENOUGH memory is the key.
Remember, to EXPAND the hash takes 1.5x the final size - the original 16K
version plus the new 32K have to exist at the same time so that the data can
be copied...
What's interesting is that this goes back to my first tentative changes in
ntop, where a number of key (m)(c)(re)alloc calls don't test the return
code...
list->idx = myGlobals.device[actualDeviceId].actualHashSize;
myGlobals.device[actualDeviceId].actualHashSize *= 2; /* Double
*/
sz =
myGlobals.device[actualDeviceId].actualHashSize*sizeof(struct hostTraffic*);
>>> myGlobals.device[actualDeviceId].hash_hostTraffic = (struct
hostTraffic**)realloc(myGlobals.device[actualDeviceId].hash_hostTraffic,
sz);
memset(&myGlobals.device[actualDeviceId].hash_hostTraffic[list->idx],
0, sizeof(struct hostTraffic*)*list->idx);
traceEvent(TRACE_INFO, "Extending hash size
[newSize=%d][deviceId=%d]",
myGlobals.device[actualDeviceId].actualHashSize,
actualDeviceId);
in globals-core.h, the call becomes a macro to force it to ntop's version:
#define realloc(p,sz) ntop_saferealloc(p, sz, __FILE__, __LINE__)
extern void* ntop_saferealloc(void* ptr, unsigned int sz, char*
file, int line);
I had a re-write of that code, based on my own pmmm library, that I haven't
dusted off for months.
Before I chase snipes, can you uncomment these lines in leaks.c (just delete
the #ifdef and #endif) and add the 2nd trace event...
/* Courtesy of Wies-Software <[EMAIL PROTECTED]> */
#undef realloc /* just to be safe */
void* ntop_saferealloc(void* ptr, unsigned int sz, char* file, int line) {
void *thePtr;
#ifdef DEBUG
if((sz == 0) || (sz > 32768)) {
traceEvent(TRACE_WARNING, "WARNING: called realloc(%p,%u) @ %s:%d",
ptr, sz, file, line);
}
#endif
thePtr = realloc(ptr, sz);
/* Temporary add this test */
if (thePtr == NULL)
traceEvent(TRACE_WARNING, "WARNING: realloc failed, returned NULL\n");
}
And tell me if you're seeing the two WARNING messages?
Thanks!
-----Burton
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of
Michael Baird
Sent: Thursday, April 11, 2002 10:09 PM
To: [EMAIL PROTECTED]
Subject: [Ntop] Ntop Performance
I'm using ntop version 2.0.99 from 04/12/02, it now daemonizes fine, I'm
using the netflow plugin as my only data source, it continues to consume
memory until finally crashing. I'm using minimal options to launch it,
./ntop -u root -d, I've got 512MB and it gets up until about 50% before
dying. I read it keeps a hash in memory of the traffic records, it
either isn't freeing that on mine, or I'm running Ntop incorrectly, or
it has a severe memory leak, any ideas or recommendations would be
appreciated. I did get some info in syslog, about the crash, I'll paste
some of the relevant info, looks like after it hits 32768 hash size, it
goes crazy, does about 5000 lines in syslog in 30 seconds and dies.
Regards
MIKE
Apr 11 22:38:23 tc3net ntop[14417]: Started thread (9226) for network
packet sniffing on eth0.
Apr 11 22:38:23 tc3net ntop[14425]: Extending hash size
[newSize=64][deviceId=1]
Apr 11 22:38:23 tc3net ntop[14425]: Extending hash size
[newSize=128][deviceId=1]
Apr 11 22:38:24 tc3net ntop[14425]: Extending hash size
[newSize=256][deviceId=1]
Apr 11 22:38:24 tc3net ntop[14425]: Extending hash size
[newSize=512][deviceId=1]
Apr 11 22:38:26 tc3net ntop[14425]: Extending hash size
[newSize=1024][deviceId=1]
Apr 11 22:38:32 tc3net ntop[14425]: Extending hash size
[newSize=2048][deviceId=1]
Apr 11 22:38:47 tc3net ntop[14425]: Extending hash size
[newSize=4096][deviceId=1]
Apr 11 22:39:08 tc3net ntop[14426]: Resetting traffic statistics...
Apr 11 22:39:09 tc3net ntop[14425]: Extending hash size
[newSize=64][deviceId=1]
Apr 11 22:39:09 tc3net ntop[14425]: Extending hash size
[newSize=128][deviceId=1]
Apr 11 22:39:09 tc3net ntop[14425]: Extending hash size
[newSize=256][deviceId=1]
Apr 11 22:39:10 tc3net ntop[14425]: Extending hash size
[newSize=512][deviceId=1]
Apr 11 22:39:12 tc3net ntop[14425]: Extending hash size
[newSize=1024][deviceId=1]
Apr 11 22:39:17 tc3net ntop[14425]: Extending hash size
[newSize=2048][deviceId=1]
Apr 11 22:39:35 tc3net ntop[14425]: Index error
idx=2049/deviceId=1:0-2048 @ [netflowPlugin.c:184]
Apr 11 22:39:35 tc3net ntop[14425]: Index error
idx=2049/deviceId=1:0-2048 @ [pbuf.c:138]
Apr 11 22:39:35 tc3net ntop[14425]: Extending hash size
[newSize=4096][deviceId=1]
Apr 11 22:40:25 tc3net ntop[14427]: Extending hash size
[newSize=64][deviceId=0]
Apr 11 22:40:28 tc3net ntop[14425]: Extending hash size
[newSize=8192][deviceId=1]
Apr 11 22:43:15 tc3net ntop[14425]: Extending hash size
[newSize=16384][deviceId=1]
Apr 11 22:45:42 tc3net ntop[14427]: Extending hash size
[newSize=128][deviceId=0]
Apr 11 22:48:25 tc3net ntop[14422]: Purging completed in 0 sec [11 hosts
deleted]
Apr 11 22:50:17 tc3net ntop[14425]: Extending hash size
[newSize=32768][deviceId=1]
Apr 11 22:53:27 tc3net ntop[14422]: Purging completed in 0 sec [12 hosts
deleted]
Apr 11 22:58:29 tc3net ntop[14422]: Purging completed in 0 sec [19 hosts
deleted]
Apr 11 23:03:32 tc3net ntop[14422]: Purging completed in 1 sec [14 hosts
deleted]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32769/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32769/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32770/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32770/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32771/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32771/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32772/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32772/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32773/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32773/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32774/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32774/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32775/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32775/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32776/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32776/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32775/deviceId=1:0-32768 @ [netflowPlugin.c:187]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32775/deviceId=1:0-32768 @ [pbuf.c:137]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32776/deviceId=1:0-32768 @ [netflowPlugin.c:187]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32776/deviceId=1:0-32768 @ [pbuf.c:137]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32777/deviceId=1:0-32768 @ [netflowPlugin.c:184]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32777/deviceId=1:0-32768 @ [pbuf.c:138]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32777/deviceId=1:0-32768 @ [netflowPlugin.c:187]
Apr 11 23:07:05 tc3net ntop[14425]: Index error
idx=32777/deviceId=1:0-32768 @ [pbuf.c:137]
Apr 11 23:07:06 tc3net ntop[14425]: Index error
idx=32778/deviceId=1:0-32768 @ [netflowPlugin.c:187]
Apr 11 23:07:06 tc3net ntop[14425]: Index error
idx=32778/deviceId=1:0-32768 @ [pbuf.c:137]
Apr 11 23:07:06 tc3net ntop[14425]: Index error
idx=32779/deviceId=1:0-32768 @ [netflowPlugin.c:187]
etc.
etc..
_______________________________________________
Ntop mailing list
[EMAIL PROTECTED]
http://listmanager.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[EMAIL PROTECTED]
http://listmanager.unipi.it/mailman/listinfo/ntop
