Dear Alan ... I would like to share my results.
So far can not get it work.
What I did:
1 Downloaded and installed "Net-NfDump-1.25" using cpan. ok
2 Copy a nfdump generated file (this file is being used by nfsen for
graphics) ok
3 Verify file content: ok
[root@AR-LXNF01 bin]$ /usr/local/bin/nfdump -r
/usr/local/nfsen/profiles-data/asn_updated/nfcapd.201802201310 | more
Date first seen Duration Proto Src IP Addr:Port
Dst IP Addr:Port Packets Bytes Flows
2018-02-20 12:54:35.741 0.000 TCP 187.118.18.149:46938 ->
169.60.79.184:5222 500 155500 1
2018-02-20 12:54:35.741 0.000 TCP 186.183.22.205:443 ->
187.118.11.18:54343 500 750000 1
2018-02-20 12:54:35.742 0.000 UDP 187.174.42.129:3658 ->
187.12.187.134:3658 500 33500 1
2018-02-20 12:54:35.745 0.000 UDP 187.118.10.47:55796 ->
186.183.22.206:443 500 32000 1
.....trunked
4 Apply perl script for asn update: ok
/usr/src/Net-NfDump-1.25/bin/nfasnupd nfcapd.201802201310
Subroutine main::pack_sockaddr_in6 redefined at
/usr/share/perl5/vendor_perl/Exporter.pm line 66.
at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8.
Subroutine main::unpack_sockaddr_in6 redefined at
/usr/share/perl5/vendor_perl/Exporter.pm line 66.
at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8.
Subroutine main::sockaddr_in6 redefined at
/usr/share/perl5/vendor_perl/Exporter.pm line 66.
at /usr/src/Net-NfDump-1.25/bin/nfasnupd line 8.
2018-02-20.13:23:28[18171]: Checking BGP database for new version.
2018-02-20.13:23:28[18171]: Loading AS database.
2018-02-20.13:23:34[18171]: Updating records.
2018-02-20.13:23:39[18171]: Processed 394412 flows in 5 secs.
4 Verify converted file content NOK
[root@AR-LXNF01 asn_updated]$ /usr/local/bin/nfdump -r nfcapd.201802201310
Skip unknown record type 10
Skip unknown record type 10
Skip unknown record type 10
...trunked
############################After that I did:
1 compiled nfdump ver 1.6.16
2 Read the file again:
[leo@arch nfdump]$ nfdump -r nfcapd.201802201310 | more
Date first seen Duration Proto Src IP Addr:Port
Dst IP Addr:Port Packets Bytes Flows
2018-02-20 13:09:03.614 27.624 UDP 187.183.47.57:56084 ->
187.183.22.205:443 1500 105500 1
2018-02-20 13:09:10.518 28.752 TCP 198.38.124.206:80 ->
181.118.2.92:50632 4500 6.6 M 1
2018-02-20 13:09:31.239 0.000 TCP 187.183.37.208:49822 ->
172.217.28.163:80 500 26000 1
2018-02-20 13:09:21.275 19.923 TCP 187.183.22.204:443 ->
187.183.21.174:6036 1500 2.2 M 1
As you can see with this newer version of nfdump I can read the file but
don't see the asn number.
############################ some debug
While reading the nfasnupd script found the following important paths:
my $DBDIR = "/var/db/flowtools";
my $CASNSDB = "/usr/local/etc/asns.txt"; # cuscom ASN
DB #This file was not here, I copied it from
/usr/src/Net-NfDump-1.25/bin/asns.txt
my $BGPDB = "/var/tmp/asns-$>.gz";
#this file does not exist even while script is working.
my $BGPDB_MD5 = "/var/tmp/asns-$>.gz.md5"; #this file does not exist
even while script is working.
my $GEO_DB4 = $DBDIR.'/geo/GeoIP.dat'; #this file does not exist
even while script is working.
my $GEO_DB6 = $DBDIR.'/geo/GeoIPv6.dat'; #this file does not exist
even while script is working.
my $FNAME_TMP = ".nffile_$$.tmp";
my $DEBUG = 1;
my $FNAME;
my $VERSION = "1.10";
###############So ...
I think those files are very important for the script to work propperly.
You you confirm if you can see those files ?
Also can you share a few lines of the nfdump output when displays
properly the asn number ?
btw ... my generated file is bigger than original.
Ok .... thats all ... thanks for the help , any idea would be preciated.
Leandro.
On 09/02/18 03:58, Alan Whinery wrote:
On 1/26/2018 4:52 AM, Leandro wrote:
Hi, Alan , thanks for the response.
It think the most difficult part of this is how to insert asn number
on the flow data structure.
First I need to read the data file , then iterate over all flows
(perhaps using pcap library ?) then take the ips involved in the
flow and insert their asn.
The idea is that this modification allow us to keep using nfsen
graphics and filter for further analisys.
I think it is necessary to have a very deep knowledge of the necessary
development tools.
Do you know some tool / api to analize and edit the flow data file at
a higher level ?
After an uncanny sequence of events, I do, in fact have an answer to this.
We have a flow analysis project going and not long after your question,
I was tasked with doing the very same thing here. It turns out that two
of our border routers are Juniper QFX, which appear only to export
SFlow, and they don't write ASNs in the records, even though they have
BGP tables.
I found a Perl module https://metacpan.org/pod/Net::NfDump which has the
necessary capabilities. While I was in the process of getting something
working to insert AS numbers into flow files, I found that the example
program, "nfasnupd" not only inserts AS numbers, but also geo info,
which I had thought about as a next step!
Command updates nfdump file and adds AS and geoIP information
Usage:
nfasnupd [ -d <level> ] -b -g [ -a -5 -4 -6 ] [ -c <ASN_db_file> ]
<nfdump_file>
Options:
-d <level> : debug level (dafault: 1)
-B do NOT update AS numbers (srcas, dstas)
-g update country code (*xsrcport, *xdstport)
-a <file> : path to BGPDB file (default: /var/tmp/asns-0.gz)
-5 <file> : path to BGPDB MD5 (default: /var/tmp/asns-0.gz.md5)
-c <file> : path to additional textfile with ASN mapping (default:
/usr/local/etc/asns.txt)
-4 <file> : path to IPv4 GeoIP database (default:
/var/db/flowtools/geo/GeoIP.dat)
-6 <file> : path to IPv6 GeoIP database (default:
/var/db/flowtools/geo/GeoIPv6.dat)
Part of libnf.net project, version: 1.10
I am only at the stage of running it on sample files, not yet committing
to bulk operations, but it is promising.
It does make the flow file smaller. I haven't yet figured out why.
If you hand it a flow file, it effectively rewrites it in place
(probably writes a temp and then moves it to the original).
the Geo stuff doesn't happen by default, so one only has to complete
dependencies and run
./nfasnupd flowfile.nfdump
If you're not Perl-proficient, send me an off list message. I'm going to
try to move toward bulk conversion, with appropriate backups first.
-Alan
Thanks,
Leo.
On 26/01/18 00:08, Alan Whinery wrote:
Of course, generally when you export flows from a BGP router with a full
table, it should already have ASNs populated.
If you have flow data with no ASN, probably the easiest way to fill it
in would be to script something with MaxMind's open source ASN data:
https://www.maxmind.com/en/open-source-data-and-api-for-ip-geolocation
I don't know off-hand of software that updates fields in nfdump files,
but there must be something out there, or some Perl or Python modules to
do so.
In the past, I've rolled my own ASN-to-prefix cross-ref by grabbing the
global routing table from a BGP router and then annotating it with the
asn lists from cidr-report.org:
http://www.cidr-report.org/as2.0/autnums.html
which is linked from:
http://www.cidr-report.org/as2.0/
On 1/25/2018 5:37 AM, Leandro wrote:
Hi guys , Im trying to analyze incoming traffic from an specific asn ,
I can not filter this using source ip since this operator uses a lot
of subnets (about 7k).
My idea is to grab a flow file and insert the asn for further
analysis. Is there something about this ?
Any idea would help ,
Regards ,
Leo.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
