Hi Peter, today it happened again.
I did not follow your unicorn attempt (I am not a girl ;-) but this time I managed to save the nfcapd-file (2.4 GBytes). My nfdump (Version: NSEL-NEL1.6.22) will neither stop nor end when I try to get IPv6 top talkers. It was a scan precisely aimed on routed networks in our IPv6 /29 (non-routed were not touched!) searching for SIP devices from some jerks host by Digital Ocean. Interested in the file? Best regards, Jens Am 18.08.21 um 10:16 schrieb Peter Haag:
Hi Jens, Actually this is surprising. I now, that nfdump has it's limits but several hours is a bit too much. If you have time to experiment, I would like you to test/compare the 1.7 beta release. If this is still an issue, I would like to dig into this more deeply. You can do: # Checkout unicorn branch: $ git clone -b unicorn https://github.com/phaag/nfdump.git master.unicorn Build the unicorn branch as you used to do. A minimal cd master.unicorn sh bootstrap; ./configure; make should do the trick. In order not to pollute your original installation, do not (yet:) run make install. create a testdir withing the master.unicorn mkdir flows Although nfdump-1.7 reads old files transparently it may be fair to convert them first and do the tests later. Convert each of the files: ./nfdump -r /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090755 -y -w flows/nfcapd.202108090755 ... Now try to run the tests with the top talkers: ./nfdump -r flows/nfcapd.xxx -s ... and compare. Feedback on the results would be appreciated. You could also run the top talker directly on the original files, but this would introduce a small overhead for converting. You may contact me also off list, if something does not work as expected. You find my email in the AUTHORS file. Thanks und Gruss - Peter On 09.08.21 21:18, Jens Hektor wrote:Am 09.08.21 um 16:06 schrieb Jens Hektor:Maybe it is not 2GB related, I am looking into the IPv6 flows ...Having switched to the cli nfdump I now believe that nfdump does not performan as one is used when it comes to *heavy* IPv6 flows. Particularly I try to look at top talkers of these files, especially in the "inet6" domain: -rw-r--r--. 1 apache apache 3,5G 9. Aug 08:00 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090755 -rw-r--r--. 1 apache apache 2,3G 9. Aug 08:04 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090800 -rw-r--r--. 1 apache apache 891M 9. Aug 08:10 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090805 -rw-r--r--. 1 apache apache 702M 9. Aug 08:15 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090810 -rw-r--r--. 1 apache apache 674M 9. Aug 08:20 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090815 -rw-r--r--. 1 apache apache 737M 9. Aug 08:25 /usr/local/nfsen/profiles-data/live/ixia-poc/2021/08/09/nfcapd.202108090820 2021/08/09/nfcapd.202108090820: Sys: 36.063s 2021/08/09/nfcapd.202108090815: Sys: 38.893s 2021/08/09/nfcapd.202108090810: Sys: 35.795s 2021/08/09/nfcapd.202108090805: Sys: 6141.546s 2021/08/09/nfcapd.202108090800: - still waiting (started 3 hours ago) Background: we have "research scanners" in our university network <off> something you really don't want ;-) </off> My guess is that they started with IPv6 scanning today but I would need an according output from the netflows to be sure. They stopped around 08:05 as one can guess from the size of the flow data. nfsen talks about 200 kflows/s but my guess is that they overdrove pretty much everything of the infrastructure I have. Have not seen this effect with IPv4 before at similar rates. There will be some clear words with the "researchers" the next days ;-) Anyway, facit: nfdump seems to be less effectice with IPv6 to me. Any confirms? Best regards, Jens P.S. This is nothing that astonishes me as there are 128 bits to process compared to the well known 32 bits. P.P.S. So not a 2GB issue. _______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
