Well, there are lots of other places to "use up" memory, esp w/ 32K hosts and ghu knows how many sessions.
Seriously, how much memory are we talking about? On my network, ntop uses about 8% of 256MB (call it 20-21MB) and that's for about 6 machines, most inactive, with about 24 hosts contacted... Each host uses up ~2K, each protocol takes up space in each host entry, etc. (Check ntop.h for HostTraffic) You might #define MEMORY_DEBUG - but I don't know if that version will even compile -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Michael Baird Sent: Tuesday, April 30, 2002 2:58 PM To: [EMAIL PROTECTED] Subject: RE: [Ntop] Ntop 29-4-02 snapshot Yea, I tried that patch, didn't notice anything different, still used up all the memory on the machine, after 3 hours. Regards MIKE On Tue, 2002-04-30 at 06:09, Burton M. Strauss III wrote: > That message is not - necessarily - indicative of a problem. (code is in hash.c around 840). > > What it's doing is walking the list to find the right entry. > > while(list != NULL) { > if(list->idx == hostIdx) { > allRight = 1; > break; > } else { > prevList = list; > list = list->next; > } > } > > There could be other reasons (?) for this. But I'm suspicious because of the proximity. > > Have you tried the patch? If you ARE getting the out of memory problem, then that could explain it. If you're not, then the > problem is going to take another approach to get the data to figure out what's up... > > -----Burton > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of > Michael Baird > Sent: Monday, April 29, 2002 11:35 AM > To: [EMAIL PROTECTED] > Subject: RE: [Ntop] Ntop 29-4-02 snapshot > > > Burton, I get bunch of these messages as well along the way, maybe this > could be related. > > 9/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32746) failed [host not found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32747) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32748) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32749) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32750) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32751) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32752) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32753) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32754) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32755) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32756) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32757) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32758) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32759) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32760) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32761) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32762) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32763) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32764) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32765) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32766) failed [host not > found] > 29/Apr/2002 12:37:01 ERROR: purgeHostIdx(1,32767) failed [host not > > Regards > MIKE > > > On Mon, 2002-04-29 at 12:20, Burton M. Strauss III wrote: > > First off, -s has been removed as -s <hashsize> > > > > It's now a (future) flag to turn off promiscuous mode. That is the code to set it is in: > > > > initialize.c: 980 myGlobals.disablePromiscuousMode == 1 ? 0 : 1, > > main.c: 452 myGlobals.disablePromiscuousMode = 1; > > globals.h: 105 u_char disablePromiscuousMode; > > > > But that's all. > > > > AFAIK, there is now no limit on hash size growth. The constant is still defined, but it's not used anymore: > > > > ntop.h: 1311 #define MAX_HASH_SIZE 32768 > > > > The code that resizes the hash is this: > > > > if(!hostFound) { > > int sz; > > > > list->idx = myGlobals.device[actualDeviceId].actualHashSize; > > if(myGlobals.device[actualDeviceId].actualHashSize < 512) > > myGlobals.device[actualDeviceId].actualHashSize = 512; > > else > > myGlobals.device[actualDeviceId].actualHashSize *= 2; /* Double */ > > sz = myGlobals.device[actualDeviceId].actualHashSize*sizeof(struct hostTraffic*); > > myGlobals.device[actualDeviceId].hash_hostTraffic = (struct > > hostTraffic**)realloc(myGlobals.device[actualDeviceId].hash_hostTraffic, sz); > > memset(&myGlobals.device[actualDeviceId].hash_hostTraffic[list->idx], > > 0, sizeof(struct hostTraffic*)*(myGlobals.device[actualDeviceId].actualHashSize-list->idx)); > > traceEvent(TRACE_INFO, "Extending hash size [newSize=%d][deviceId=%d]", > > myGlobals.device[actualDeviceId].actualHashSize, > > actualDeviceId); > > } else > > > > The only limit is available memory. Remember that the realloc will - under the covers - require enough memory for the new size > > (rounded to a full page) + the original size (it does a malloc, copy, free set internally). > > > > Now there is a report that the realloc can fail and cause all kinds of horrible problems (and I agree that should be tested for), > > but if you have enough ram, you should be able to grow beyond 32K... > > > > You can TRY the attached patch. It's under development, and will ONLY report the error if the realloc fails!! > > > > (All greps are from 2.0.99 ntop-02-04-27.tgz) > > > > -----Burton > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of > > Michael Baird > > Sent: Monday, April 29, 2002 10:35 AM > > To: [EMAIL PROTECTED] > > Subject: [Ntop] Ntop 29-4-02 snapshot > > > > > > I'm using ntop with the netflow plugin, and it's finally starting to > > look promising, except for a couple of issues. It now compiles > > properly, runs properly, and works properly, however it doesn't clear > > the hash in RAM, my understanding was at 32768 it should clear the hash > > and start over, is this incorrect, or do I need to run it with some > > extra arguments, I tried the -S 1 and -S 2 options, neither work with > > the netflow plugin, -S 0 does. I'm currently using /usr/local/bin/ntop > > -u root -d as my only arguments, I've tried -s <hashsize> as described > > in the FAQ but ntop doesn't pay attention to this either. > > > > Regards > > MIKE > > > > > > > > _______________________________________________ > > Ntop mailing list > > [EMAIL PROTECTED] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > _______________________________________________ > Ntop mailing list > [EMAIL PROTECTED] > http://listgateway.unipi.it/mailman/listinfo/ntop > > _______________________________________________ > Ntop mailing list > [EMAIL PROTECTED] > http://listgateway.unipi.it/mailman/listinfo/ntop > _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop
