Hi Jens, Actually this is surprising. I now, that nfdump has it's limits but several hours is a bit too much.
If you have time to experiment, I would like you to test/compare the 1.7 beta release. If this is still an issue, I would like to dig into this more deeply. You can do: # Checkout unicorn branch: $ git clone -b unicorn https://github.com/phaag/nfdump.git master.unicorn Build the unicorn branch as you used to do. A minimal cd master.unicorn sh bootstrap; ./configure; make should do the trick. In order not to pollute your original installation, do not (yet:) run make install. create a testdir withing the master.unicorn mkdir flows Although nfdump-1.7 reads old files transparently it may be fair to convert them first and do the tests later. Convert each of the files: ./nfdump -r /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090755 -y -w flows/nfcapd.202108090755 ... Now try to run the tests with the top talkers: ./nfdump -r flows/nfcapd.xxx -s ... and compare. Feedback on the results would be appreciated. You could also run the top talker directly on the original files, but this would introduce a small overhead for converting. You may contact me also off list, if something does not work as expected. You find my email in the AUTHORS file. Thanks und Gruss - Peter On 09.08.21 21:18, Jens Hektor wrote: > Am 09.08.21 um 16:06 schrieb Jens Hektor: >> Maybe it is not 2GB related, I am looking into the IPv6 flows ... > > Having switched to the cli nfdump I now believe > that nfdump does not performan as one is used > when it comes to *heavy* IPv6 flows. > > Particularly I try to look at top talkers of these files, > especially in the "inet6" domain: > > -rw-r--r--. 1 apache apache 3,5G 9. Aug 08:00 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090755 > -rw-r--r--. 1 apache apache 2,3G 9. Aug 08:04 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090800 > -rw-r--r--. 1 apache apache 891M 9. Aug 08:10 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090805 > -rw-r--r--. 1 apache apache 702M 9. Aug 08:15 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090810 > -rw-r--r--. 1 apache apache 674M 9. Aug 08:20 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090815 > -rw-r--r--. 1 apache apache 737M 9. Aug 08:25 > /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090820 > > 2021/08/09/nfcapd.202108090820: Sys: 36.063s > 2021/08/09/nfcapd.202108090815: Sys: 38.893s > 2021/08/09/nfcapd.202108090810: Sys: 35.795s > 2021/08/09/nfcapd.202108090805: Sys: 6141.546s > 2021/08/09/nfcapd.202108090800: - still waiting (started 3 hours ago) > > Background: > > we have "research scanners" in our university network > <off> something you really don't want ;-) </off> > My guess is that they started with IPv6 scanning today > but I would need an according output from the netflows > to be sure. > > They stopped around 08:05 as one can guess from > the size of the flow data. > > nfsen talks about 200 kflows/s but my guess is that they > overdrove pretty much everything of the infrastructure I have. > > Have not seen this effect with IPv4 before at similar rates. > > There will be some clear words with the "researchers" the next days ;-) > > Anyway, facit: nfdump seems to be less effectice with IPv6 to me. > > Any confirms? > > Best regards, Jens > > P.S. This is nothing that astonishes me as there are 128 bits to process > compared to the well known 32 bits. > > P.P.S. So not a 2GB issue. > > > _______________________________________________ > Nfsen-discuss mailing list > Nfsen-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > -- Be nice to your netflow data. Use NfSen and nfdump :) _______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
