Daniel thanks for your comments. The first answer to your question is to read the code :-)
We had some memory leaks in the past but I;m not aware of any of them recently. Valgrind confirms that. The memory is allocated and released during flow life time thus at the end of its life the memory is freed. Unix generally reports the top memory usage, so sometimes you read the top memory being allocated. Concerning hashing I am taking advantage of your suggestions and currently working at that. Finally, I have made quite some changes in nprobe 6.4.x with respect to the version you use. You can read them here http://www.ntop.org/blog/nprobe/tuning-nprobe-6-4-scalability-and-performance/. I suggest you to update whenever possible (note that the hashing function is under test thus it is not included in the code being released). Cheers Luca On Apr 28, 2011, at 11:19 AM, Daniel Aschwanden wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi all, Hi Luca, > > I have a further question regarding the memory management of nprobe: > > How is the memory management handled? > > I'm asking that question, because I'm experiencing the following: > > Firstly, nprobe is started and it consumes only about one percent of > memory, because there isn't any severe traffic. Then I'm starting my > traffic generators which generates around 100k flows per second (as > Maurizio had outlined in the previous mail). This leads of course to a > bigger memory usage. However, if the traffic generation has finished and > there is no traffic again, the memory consumption stays as it was. > > So my question: why does nprobe not freeing the unused memory? > > I guess this memory growth is according to longer bucket lists, since > the hash size stays constant. As mentioned before, I'm also experiencing > very long lists (19-31 max. bucket search). > > Furthermore, if I run the traffic generation for a very long time, > nprobes memory consumption is monotonically growing at a low pace till > it runs out of memory. > > Some details of the configuration: > nprobe v.6.3.0 ($Revision: 1947 $) for i686-pc-linux-gnu > - -t=10 > - -d=10 > - -l=1 > - -s=2 > - -V=5 > - -w=8388608 > - -M=8000000 > - -e=0 > > Thanks for your comments and explanation of that. > > Cheers, > Dani > > > On 04/20/2011 10:20 AM, Maurizio Molina wrote: >> Hi Luca, >> our tests are trying to put as much stress as we can on nprobe from a >> flows-per-second point of view, and since we don't have dedicated HW >> generators we generate 1 packet flows, TCP, of the smallest size >> possible. The flows/s (and pk/s as well, since flows are 1 pkts long >> flows) are in these tests 100K/s. There are six virtual machines >> generating this traffic, so 6 different source IPs. Each of them chooses >> randomly a destination port Xi (i=1,....,6), then generates 60,000 >> packets incrementing each time deterministically the destination port by >> one (Di, Di+1, Di+2, ...), then the loop repeats itself with every VM >> incrementing deterministically Xi by one. >> So, at the end, two nested loops on the src and dst ports. This means >> actually low "entropy" (a lot of subsequent pkt headers will differ by >> one bit only) but this is similar to what happens e.g. in a port >> scanning. A "good" hash should distribute randomly also with 1 bit >> variations, but I know this is difficult. What hash is nprobe using, BTW? >> Here below, the probability of long lists that we expect with a >> perfectly uniform hash distribution for that load, hash size of 2^23, >> flows kept in the flow cache for 11 seconds (10 sec idle tout and 1 sec >> scan cycle). Bucket sizes of 16 elements are clearly indicating an >> uneven distribution of the hash. >> Cheers, >> Maurizio >> >> K P(list>k) >> 0 1.00E+00 >> 1 1.16E-01 >> 2 7.54E-03 >> 3 3.30E-04 >> 4 1.08E-05 >> 5 2.83E-07 >> 6 6.19E-09 >> 7 1.16E-10 >> 8 1.90E-12 >> 9 2.77E-14 >> 10 3.63E-16 >> 11 4.33E-18 >> 12 4.73E-20 >> 13 4.77E-22 >> 14 4.47E-24 >> 15 3.91E-26 >> 16 3.20E-28 >> 17 2.47E-30 >> 18 1.80E-32 >> 19 1.24E-34 >> 20 8.15E-37 >> >> >> >> On 19.04.2011 18:07, Luca Deri wrote: >>> Daniel >>> it looks your traffic is not easy balance-able then. What kind of >>> packets are you using? >>> >>> Luca >>> > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > - -- > daniel aschwanden > junior engineer > > open systems ag > raeffelstrasse 29 > ch-8045 zurich > t: +41 44 455 74 96 > f: +41 44 455 74 01 > [email protected] > > http://www.open.ch > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iF4EAREIAAYFAk25MSEACgkQxIzmH53Qg6EQSQEAghNTyKmejaAjZdhiI/25xrFV > Ar3F3tUuRcgtW0OTSUgA/2c/npNbNQa8SASgW3lsHEnSJIZGER66QhZEh06o9wtx > =Iy93 > -----END PGP SIGNATURE----- > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc --- We can't solve problems by using the same kind of thinking we used when we created them - Albert Einstein _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
