Hi Paolo,

the config is now simplified as you suggested, but for me it seems
that the bgp attributes are still not merged with the netflow. In
order to make sure, that the peers are up and running when netflow
hits nfacctd, I waited 18+ hours before checking the logs.

Thanks,

Tamas

vargat@noc-netflow-aggr:/tmp$ sudo cat test_20171109-1045.txt | head -5
TAG,SRC_MAC,DST_MAC,SRC_AS,DST_AS,AS_PATH,PREF,MED,SRC_IP,DST_IP,SRC_PORT,DST_PORT,PROTOCOL,SH_COUNTRY,DH_COUNTRY,SH_POCODE,DH_POCODE,PACKETS,BYTES
0,00:00:00:00:00:00,00:00:00:00:00:00,0,0,,0,0,109.71.164.16,5.159.216.124,65153,51403,udp,LU,LU,,,1,261
0,00:00:00:00:00:00,00:00:00:00:00:00,0,0,,0,0,109.71.162.28,109.71.164.6,61524,1935,tcp,LU,LU,,,20,846
0,00:00:00:00:00:00,00:00:00:00:00:00,0,0,,0,0,109.71.161.162,149.154.157.151,443,7272,tcp,LU,IT,,06049,1,217
0,00:00:00:00:00:00,00:00:00:00:00:00,0,0,,0,0,41.230.1.145,185.13.90.76,54360,443,tcp,TN,LU,,,1,40

vargat@noc-netflow-aggr:/tmp$ sudo grep -r "\"41.230.0" bgp-*-1000.log | head -3
bgp-5_159_218_1-1000.log:{"timestamp": "2017-11-09 10:00:00",
"peer_ip_src": "5.159.218.1", "event_type": "dump", "afi": 1, "safi":
1, "ip_prefix": "41.230.0.0/16", "bgp_nexthop": "212.3.238.185",
"as_path": "3356 6762 2609", "comms": "34655:3 34655:40 34655:406
34655:4060", "origin": 0, "local_pref": 100, "med": 400}
bgp-5_159_218_2-1000.log:{"timestamp": "2017-11-09 10:00:00",
"peer_ip_src": "5.159.218.2", "event_type": "dump", "afi": 1, "safi":
1, "ip_prefix": "41.230.0.0/16", "bgp_nexthop": "212.3.238.185",
"as_path": "3356 6762 2609", "comms": "34655:3 34655:40 34655:406
34655:4060", "origin": 0, "local_pref": 100, "med": 400}
bgp-5_159_218_4-1000.log:{"timestamp": "2017-11-09 10:00:00",
"peer_ip_src": "5.159.218.4", "event_type": "dump", "afi": 1, "safi":
1, "ip_prefix": "41.230.0.0/16", "bgp_nexthop": "212.3.238.185",
"as_path": "3356 6762 2609", "comms": "34655:3 34655:40 34655:406
34655:4060", "origin": 0, "local_pref": 100, "med": 400}

debug: false
daemonize: true
nfacctd_ip: 216.172.X.X
nfacctd_port: 9966
logfile: /var/log/nfacctd.log
plugins: print[test]
!BGP
bgp_daemon: true
bmp_daemon: true
bgp_daemon_ip: 216.172.X.X
bgp_daemon_id: 216.172.X.X
bgp_peer_as_skip_subas: true
bgp_daemon_max_peers: 20
bgp_table_dump_file: /tmp/bgp-$peer_src_ip-%H%M.log
bgp_table_dump_refresh_time: 36000
bgp_follow_default: 5
bgp_agent_map: bgp_agents
nfacctd_as: bgp
pmacctd_as: bgp
nfacctd_net: bgp
pmacctd_net: bgp
bgp_aspath_radius: 3
bgp_daemon_msglog: true
nfacctd_as_new: bgp
bgp_peer_src_as_type: bgp
geoipv2_file: /usr/local/share/GeoIP/GeoLite2-City.mmdb
aggregate[test]: src_host, dst_host, src_port, dst_port, proto, tag,
src_host_country, dst_host_country, src_host_pocode, dst_host_pocode,
src_mac, dst_mac, src_as, dst_as, local_pref, med, as_path
print_refresh_time[test]: 900
print_history[test]: 15m
print_output[test]: csv
print_output_file[test]: /tmp/test_%Y%m%d-%H%M.txt
print_cache_entries[test]: 1310840
print_history_roundoff[test]: m
!fine tuning
plugin_buffer_size:16384
plugin_pipe_size:161920000
plugin_pipe_size[test]: 80240000

On Wed, Nov 8, 2017 at 3:21 PM, Paolo Lucente <pa...@pmacct.net> wrote:
>
> Hi Tamas,
>
> From your outputs definitely looks everything is in order. I wonder
> though, since you use the IMT plugin, if those entries are created
> before the BGP session is successfully established. Any chance, keeping
> things simple and for the sake of a test, you can try the same with the
> print plugin writing to flat file(s)? Failing that, i'd be glad to have
> a look at the issue myself if SSH access to the box is possible (in
> which case we can follow-up by unicast email).
>
> Paolo
>
> On Wed, Nov 08, 2017 at 12:33:32PM +0100, Varga Tamas wrote:
>> Hi Paolo,
>>
>> I have some problems getting info from bgp tables merged with netflow
>> data. Unfortunately, after going through the wiki/faq/mailling-list I
>> still can't figure out what is the missing piece from the config files
>> in order to get these feature working.
>>
>> I have checked the following things based on suggestions found on the
>> mailing list:
>> - Bgp peer ip == netflow agent IP
>> - bgp_agent_map
>> - Dump bgp tables fetched from the bgp peer
>> - Use "bgp" for (pmacctd|nfacctd)_net/as
>> - Verified bgp peering and netflow
>>
>> Thanks,
>>
>> Tamas
>>
>> Relevant debug/config info
>>
>> ### nfacctd.conf
>> !BGP
>> bgp_daemon: true
>> bmp_daemon: true
>> bgp_daemon_ip: 216.172.X.X
>> bgp_daemon_id: 216.172.X.x
>> bgp_peer_as_skip_subas: true
>> bgp_daemon_max_peers: 20
>> bgp_table_dump_file: /tmp/bgp-$peer_src_ip-%H%M.log
>> bgp_table_dump_refresh_time: 3600
>> bgp_follow_default: 2
>> bgp_agent_map: bgp_agents
>> nfacctd_as: bgp
>> pmacctd_as: bgp
>> nfacctd_net: bgp
>> pmacctd_net: bgp
>> bgp_aspath_radius: 3
>> nfacctd_as_new: bgp
>> bgp_peer_src_as_type: bgp
>> aggregate[mem]: src_host, dst_host, src_port, dst_port, proto, tag,
>> src_host_country, dst_host_country, src_host_pocode, dst_host_pocode,
>> src_as, dst_as, med, as_path
>>
>> ### bgp_agents
>> /etc/pmacct# cat bgp_agents
>> bgp_ip=5.159.218.5 ip=5.159.218.5
>>
>> ###Netflow agent
>> DEBUG ( default/core ): NfV9 agent         : 5.159.218.5:2097
>> DEBUG ( default/core ): NfV9 template type : flow
>> DEBUG ( default/core ): NfV9 template ID   : 260
>> DEBUG ( default/core ):
>> -------------------------------------------------------------
>> DEBUG ( default/core ): |    pen     |         field type         |
>> offset |  size  |
>> DEBUG ( default/core ): | 0          | in packets         [2    ] |
>>   0 |      4 |
>> DEBUG ( default/core ): | 0          | in bytes           [1    ] |
>>   4 |      4 |
>> DEBUG ( default/core ): | 0          | IPv4 src addr      [8    ] |
>>   8 |      4 |
>> DEBUG ( default/core ): | 0          | IPv4 dst addr      [12   ] |
>>  12 |      4 |
>> DEBUG ( default/core ): | 0          | input snmp         [10   ] |
>>  16 |      4 |
>> DEBUG ( default/core ): | 0          | output snmp        [14   ] |
>>  20 |      4 |
>> DEBUG ( default/core ): | 0          | last switched      [21   ] |
>>  24 |      4 |
>> DEBUG ( default/core ): | 0          | first switched     [22   ] |
>>  28 |      4 |
>> DEBUG ( default/core ): | 0          | L4 src port        [7    ] |
>>  32 |      2 |
>> DEBUG ( default/core ): | 0          | L4 dst port        [11   ] |
>>  34 |      2 |
>> DEBUG ( default/core ): | 0          | src as             [16   ] |
>>  36 |      4 |
>> DEBUG ( default/core ): | 0          | dst as             [17   ] |
>>  40 |      4 |
>> DEBUG ( default/core ): | 0          | BGP IPv4 next hop  [18   ] |
>>  44 |      4 |
>> DEBUG ( default/core ): | 0          | IPv4 src mask      [9    ] |
>>  48 |      1 |
>> DEBUG ( default/core ): | 0          | IPv4 dst mask      [13   ] |
>>  49 |      1 |
>> DEBUG ( default/core ): | 0          | L4 protocol        [4    ] |
>>  50 |      1 |
>> DEBUG ( default/core ): | 0          | tcp flags          [6    ] |
>>  51 |      1 |
>> DEBUG ( default/core ): | 0          | tos                [5    ] |
>>  52 |      1 |
>> DEBUG ( default/core ): | 0          | direction          [61   ] |
>>  53 |      1 |
>> DEBUG ( default/core ): | 0          | forwarding status  [89   ] |
>>  54 |      1 |
>> DEBUG ( default/core ): | 0          | sampler ID         [48   ] |
>>  55 |      2 |
>> DEBUG ( default/core ): | 0          | 234                [234  ] |
>>  57 |      4 |
>> DEBUG ( default/core ): | 0          | 235                [235  ] |
>>  61 |      4 |
>> DEBUG ( default/core ):
>> -------------------------------------------------------------
>> DEBUG ( default/core ): Netflow V9/IPFIX record size : 65
>>
>> ### BGP peer
>> INFO ( default/core/BGP ): [5.159.218.5] BGP peers usage: 1/50
>> INFO ( default/core/BGP ): [5.159.218.5] Capability: MultiProtocol [1]
>> AFI [1] SAFI [1]
>> INFO ( default/core/BGP ): [5.159.218.5] Capability: 4-bytes AS [41] ASN 
>> [34655]
>> INFO ( default/core/BGP ): [5.159.218.5] BGP_OPEN: Local AS: 34655
>> Remote AS: 34655 HoldTime: 180
>> DEBUG ( default/core/BGP ): [5.159.218.5] BGP_KEEPALIVE received
>> DEBUG ( default/core/BGP ): [5.159.218.5] BGP_KEEPALIVE sent
>>
>> ### pmacct
>> /etc/pmacct# pmacct -s -T flows,5 -c dst_host_country
>> TAG         SRC_AS      DST_AS      AS_PATH                  MED
>> SRC_IP           DST_IP           SRC_PORT  DST_PORT  PROTOCOL
>> SH_COUNTRY  DH_COUNTRY  SH_POCODE     DH_POCODE     PACKETS
>>    BYTES
>> 0           0           0           ^$                       0
>> 109.71.162.32    178.187.170.198  1935      4010      tcp         LU
>>        RU                        659300        2
>> 2904
>> 0           0           0           ^$                       0
>> 46.188.106.58    93.93.53.199     62090     80        tcp         RU
>>        LU          101194                      1
>> 40
>> 0           0           0           ^$                       0
>> 185.13.90.86     85.29.74.98      443       49498     tcp         LU
>>        FI                        74130         98
>> 124015
>> 0           0           0           ^$                       0
>> 93.93.51.195     5.228.17.199     80        48324     tcp         LU
>>        RU                        101194        1
>> 1280
>> 0           0           0           ^$                       0
>> 109.71.161.146   2.86.239.153     443       58337     tcp         LU
>>        GR                                      1
>> 1400
>>
>> ### Example
>>
>> Json output from bgp dump
>> {"timestamp": "2017-11-02 08:41:00", "peer_ip_src": "5.159.218.5",
>> "event_type": "dump", "afi": 1, "safi": 1, "ip_prefix":
>> "85.29.64.0/18", "bgp_nexthop": "80.81.192.144", "as_path": "6667
>> 13170", "comms": "6667:900 6667:1000 6667:2000 6667:3000 6667:4002
>> 34655:2 34655:40 34655:401 34655:2002 34655:4010", "origin": 0,
>> "local_pref": 110, "med": 206}
>>
>> pmacct output
>> 0           0           0           ^$                       0
>> 185.13.90.86     85.29.74.98      443       49498     tcp         LU
>>        FI                        74130         98
>> 124015
>>
>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to