[pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-17 Thread Grassot, Wilfrid
Good afternoon

 

I cannot have my netflow augmented with bgp data (src_as, dst_as,
peer_dst_ip.) all of the BGP data stay 0 or are empty

An output of the csv file is: 

0,0,63.218.164.15,,62.140.128.166,220.206.187.242,2123,2123,udp,1,40

 

Where 0,0 are the missing src_as, dst_as  and , , is the missing
peer_dst_ip

 

 

I try to monitor traffic of a L3VPN by having all routers sending netflow
to nfacctd and augment them with BGP data.

The nfacctd collector peers with the route-reflector on address-family
vpnv4.

Please mind the network is a confederation network with sub-as

 

I cannot figure out what is wrong

 

BGP session is up,

bgp_table_dump_file collects properly all routes from the vrf

netflow is properly collected by nfacctd

 

But all aggregate values that should augment the data stay at zero for the
AS, or empty like peer_dst_ip

 

My bgp_agent_map file has the below entry

bgp_ip=w.x.y.z.   ip=0.0.0.0/0 where w.x.y.z is the IP address of the
nfacctd collector

 

my nfacctd config file is:

 

daemonize: false

debug: true

bgp_peer_as_skip_subas: true

bgp_src_std_comm_type: bgp

bgp_src_ext_comm_type: bgp

bgp_src_as_path_type: bgp

bgp_agent_map: /usr/local/etc/pmacct/map.txt

nfacctd_as_new: bgp

nfacctd_net: bgp

nfacctd_as: bgp

 

nfacctd_port: 2055

nfacctd_templates_file: /usr/local/etc/pmacct/nfacctd-template.txt

 

nfacctd_time_new: true

 

plugin_buffer_size: 70240

plugin_pipe_size: 2024000

 

bgp_daemon: true

bgp_daemon_ip: w.x.y.z

bgp_daemon_id: w.x.y.z

bgp_daemon_max_peers: 100

bgp_table_dump_file: /var/spool/bgp-$peer_src_ip-%H%M.log

 

plugins: print

print_output_file: /var/spool/plugin.log

print_output_file_append: true

print_refresh_time: 3

print_output: cvs

aggregate: proto, src_host, src_port, dst_host, dst_port, src_as, dst_as,
peer_src_ip, peer_dst_ip

 

 

Thank you in advance

 

Wilfrid

 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-18 Thread Grassot, Wilfrid
Hi Paolo,

Thank you for your answer.

My bad in the description of the issue:
w.x.y.z is indeed the ipv4 address of the router loop0 which is also its
router-id.

Currently our setup is to iBGP peer with the router (router-id w.x.y.z) at
the address-family vpnv4.
We already filter out using route-target on the router for nfacctd  to
receive only ipv4 routes from the monitored L3VPN.
So the BGP daemon is only collecting routes of the monitored L3VPN

On nfacctd collector we also receive only the netflow from routers
interfaces configured on this vrf.
If I manually make the correlation of the captured netflow, I can see in
the BGP dump files the corresponding src_as, dest_as, peer_dst_ip

So netflow and BGP are fine and bgp_agent_map file is  bgp_ip=w.x.y.z.
ip=0.0.0.0/0 where w.x.y.z is the loopback0 (router-id) of the router,
and nfacctd is peering with it (sorry again for the mishap).

I use the latest pmacctd 1.7.4 and I compile with ./configure
--enable-jansson  (--enable-threads is not available)

And yes our network is a confederation of 6 sub_as.

Thank you

Wilfrid Grassot






-Original Message-
From: Paolo Lucente 
Sent: Monday, 18 May 2020 16:30
To: pmacct-discussion@pmacct.net; Grassot, Wilfrid

Subject: Re: [pmacct-discussion] BGP correlation not working with nfacctd,
all BGP set to 0


Hi Wilfrid,

Thanks for getting in touch. A couple of notes:

1) if you are sending vpnv4 routes - and if that is a requirement - then
you will need a flow_to_rd_map to map flows to the right VPN (maybe basing
on the input interface at the ingress router? just an idea);

2) Confederations always do add up to the fun :-) I may not have the
complete info at the moment in order to comment further on this;

3) bgp_ip in the bgp_agent_map may have been set incorrectly; in the
comment you say "where w.x.y.z is the IP address of the nfacctd collector"
but, according to docs, it should be set to the "IPv4/IPv6 session address
or Router ID of the BGP peer.".

You may start working on #1 and #3. Probably more info is needed for #2
and for this reason I suggest that, if things do not just work out at this
round, we move the conversation to unicast email.

Paolo


On 17/05/2020 16:24, Grassot, Wilfrid wrote:
> Good afternoon
>
> I cannot have my netflow augmented with bgp data (src_as, dst_as,
> peer_dst_ip…) all of the BGP data stay 0 or are empty
>
> An output of the csv file is:
>
> 0,0,63.218.164.15,,62.140.128.166,220.206.187.242,2123,2123,udp,1,40
>
> Where 0,0 are the missing src_as, dst_as  and , , is the missing
> peer_dst_ip
>
> I try to monitor traffic of a L3VPN by having all routers sending
> netflow to nfacctd and augment them with BGP data.
>
> The nfacctd collector peers with the route-reflector on address-family
> vpnv4.
>
> _Please mind the network is a confederation network with sub-as_
>
> __
>
> I cannot figure out what is wrong
>
> __
>
> BGP session is up,
>
> bgp_table_dump_file collects properly all routes from the vrf
>
> netflow is properly collected by nfacctd
>
> But all aggregate values that should augment the data stay at zero for
> the AS, or empty like peer_dst_ip
>
> My bgp_agent_map file has the below entry
>
> bgp_ip=w.x.y.z.   ip=0.0.0.0/0     where w.x.y.z is the IP address of
> the nfacctd collector
>
> my nfacctd config file is:
>
> daemonize: false
>
> debug: true
>
> bgp_peer_as_skip_subas: true
>
> bgp_src_std_comm_type: bgp
>
> bgp_src_ext_comm_type: bgp
>
> bgp_src_as_path_type: bgp
>
> bgp_agent_map: /usr/local/etc/pmacct/map.txt
>
> nfacctd_as_new: bgp
>
> nfacctd_net: bgp
>
> nfacctd_as: bgp
>
> nfacctd_port: 2055
>
> nfacctd_templates_file: /usr/local/etc/pmacct/nfacctd-template.txt
>
> nfacctd_time_new: true
>
> plugin_buffer_size: 70240
>
> plugin_pipe_size: 2024000
>
> bgp_daemon: true
>
> bgp_daemon_ip: w.x.y.z
>
> bgp_daemon_id: w.x.y.z
>
> bgp_daemon_max_peers: 100
>
> bgp_table_dump_file: /var/spool/bgp-$peer_src_ip-%H%M.log
>
> plugins: print
>
> print_output_file: /var/spool/plugin.log
>
> print_output_file_append: true
>
> print_refresh_time: 3
>
> print_output: cvs
>
> aggregate: proto, src_host, src_port, dst_host, dst_port, src_as,
> dst_as, peer_src_ip, peer_dst_ip
>
> Thank you in advance
>
> Wilfrid
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Grassot, Wilfrid
Hi Paolo,

Could the issue be that correlation does not work because for each
"ip_prefix" there is not one, but two or three routes collected by pmbgpd
?
Indeed because of redundancies, each prefixes are received by several
different routers in our network and by design each of the routers use
different route distinguisher (rd).
Hence the pmbgpd does not receive a unique route corresponding to best
path selected by the route-reflector, but the two or three different vpnv4
addresses (rd:a.b.c.d) corresponding to ip_prefix = a.b.c.d ?

Wilfrid


-Original Message-----
From: Grassot, Wilfrid 
Sent: Monday, 18 May 2020 17:05
To: Paolo Lucente ; pmacct-discussion@pmacct.net
Subject: RE: [pmacct-discussion] BGP correlation not working with nfacctd,
all BGP set to 0

Hi Paolo,

Thank you for your answer.

My bad in the description of the issue:
w.x.y.z is indeed the ipv4 address of the router loop0 which is also its
router-id.

Currently our setup is to iBGP peer with the router (router-id w.x.y.z) at
the address-family vpnv4.
We already filter out using route-target on the router for nfacctd  to
receive only ipv4 routes from the monitored L3VPN.
So the BGP daemon is only collecting routes of the monitored L3VPN

On nfacctd collector we also receive only the netflow from routers
interfaces configured on this vrf.
If I manually make the correlation of the captured netflow, I can see in
the BGP dump files the corresponding src_as, dest_as, peer_dst_ip

So netflow and BGP are fine and bgp_agent_map file is  bgp_ip=w.x.y.z.
ip=0.0.0.0/0 where w.x.y.z is the loopback0 (router-id) of the router,
and nfacctd is peering with it (sorry again for the mishap).

I use the latest pmacctd 1.7.4 and I compile with ./configure
--enable-jansson  (--enable-threads is not available)

And yes our network is a confederation of 6 sub_as.

Thank you

Wilfrid Grassot






-Original Message-
From: Paolo Lucente 
Sent: Monday, 18 May 2020 16:30
To: pmacct-discussion@pmacct.net; Grassot, Wilfrid

Subject: Re: [pmacct-discussion] BGP correlation not working with nfacctd,
all BGP set to 0


Hi Wilfrid,

Thanks for getting in touch. A couple of notes:

1) if you are sending vpnv4 routes - and if that is a requirement - then
you will need a flow_to_rd_map to map flows to the right VPN (maybe basing
on the input interface at the ingress router? just an idea);

2) Confederations always do add up to the fun :-) I may not have the
complete info at the moment in order to comment further on this;

3) bgp_ip in the bgp_agent_map may have been set incorrectly; in the
comment you say "where w.x.y.z is the IP address of the nfacctd collector"
but, according to docs, it should be set to the "IPv4/IPv6 session address
or Router ID of the BGP peer.".

You may start working on #1 and #3. Probably more info is needed for #2
and for this reason I suggest that, if things do not just work out at this
round, we move the conversation to unicast email.

Paolo


On 17/05/2020 16:24, Grassot, Wilfrid wrote:
> Good afternoon
>
> I cannot have my netflow augmented with bgp data (src_as, dst_as,
> peer_dst_ip…) all of the BGP data stay 0 or are empty
>
> An output of the csv file is:
>
> 0,0,63.218.164.15,,62.140.128.166,220.206.187.242,2123,2123,udp,1,40
>
> Where 0,0 are the missing src_as, dst_as  and , , is the missing
> peer_dst_ip
>
> I try to monitor traffic of a L3VPN by having all routers sending
> netflow to nfacctd and augment them with BGP data.
>
> The nfacctd collector peers with the route-reflector on address-family
> vpnv4.
>
> _Please mind the network is a confederation network with sub-as_
>
> __
>
> I cannot figure out what is wrong
>
> __
>
> BGP session is up,
>
> bgp_table_dump_file collects properly all routes from the vrf
>
> netflow is properly collected by nfacctd
>
> But all aggregate values that should augment the data stay at zero for
> the AS, or empty like peer_dst_ip
>
> My bgp_agent_map file has the below entry
>
> bgp_ip=w.x.y.z.   ip=0.0.0.0/0     where w.x.y.z is the IP address of
> the nfacctd collector
>
> my nfacctd config file is:
>
> daemonize: false
>
> debug: true
>
> bgp_peer_as_skip_subas: true
>
> bgp_src_std_comm_type: bgp
>
> bgp_src_ext_comm_type: bgp
>
> bgp_src_as_path_type: bgp
>
> bgp_agent_map: /usr/local/etc/pmacct/map.txt
>
> nfacctd_as_new: bgp
>
> nfacctd_net: bgp
>
> nfacctd_as: bgp
>
> nfacctd_port: 2055
>
> nfacctd_templates_file: /usr/local/etc/pmacct/nfacctd-template.txt
>
> nfacctd_time_new: true
>
> plugin_buffer_size: 70240
>
> plugin_pipe_size: 2024000
>
> bgp_daemon: true
>
> bgp_daemon_ip: w.x.y.z
>
> bgp_daemon_id: w.x.y.z
>
> bgp_daemon_max_peers: 100
>
> bgp_table_dump_file: /var/spool/bgp-$peer_src_ip-%

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Grassot, Wilfrid
Hi Paolo,

Unless I misunderstand the flow_to_rd_map, but this one would not help in
our case.
Indeed we already capture all the flows matching the different rd because
of our netflow setup.
nfacctd already receives only flows from the specific RDs involved in the
monitored L3VPN .

My concern is about the correlation of a flow to a src_as, dst_as,
dst_peer... retrieved from the captured BGP RIB  of this L3VPN and dumped
to  bgp-$peer_src_ip-%H%M.log,

Would you please confirm that the flow, BGP correlation can only work only
if router only advertise to the pmbgpd the best path

In other words, would you please confirm that our setup is not supported
because for each prefixes there are not a unique entry in the captured BGP
RIB, but at least 2 or 3 entries (no best path selection at the
route-reflector because each vpnv4 address are seen as unique because of
the different RDs involved) ?

Please see below an example of the pmbgpd dump file:

sudo jq '. | select(."ip_prefix" | contains ("a.b.c.d"))'
bgp-w_x_y_z-0715.log | more
{
  "seq": 3,
  "timestamp": "2020-05-19 07:15:00",
  "peer_ip_src": "w.x.y.z",
  "ip_prefix": "a.b.c.d/27",
  "rd": "0:ASN:900290024",
  "label": "63455"
}
{
  "seq": 3,
  "timestamp": "2020-05-19 07:15:00",
  "peer_ip_src": " w.x.y.z ",
  "ip_prefix": "a.b.c.d/27",
  "rd": "0:ASN:911790015",
  "label": "49061"
}
{
  "seq": 3,
  "timestamp": "2020-05-19 07:15:00",
  "peer_ip_src": " w.x.y.z ",
  "ip_prefix": "a.b.c.d/27",
  "rd": "0:ASN:911790023",
  "label": "49059"
}

Thank you
Wilfrid

-Original Message-
From: Paolo Lucente 
Sent: Tuesday, 19 May 2020 16:01
To: Grassot, Wilfrid 
Cc: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] BGP correlation not working with nfacctd,
all BGP set to 0


Hi Wilfrid,

This is very possibly point #1 of my previous email. The need for a
flow_to_rd_map to associate flows to the right RD. You can find some
examples here on how to compose it:

https://github.com/pmacct/pmacct/blob/1.7.5/examples/flow_to_rd.map.exampl
e

Paolo

On Tue, May 19, 2020 at 08:17:44AM +, Grassot, Wilfrid wrote:
> Hi Paolo,
>
> Could the issue be that correlation does not work because for each
> "ip_prefix" there is not one, but two or three routes collected by
> pmbgpd ?
> Indeed because of redundancies, each prefixes are received by several
> different routers in our network and by design each of the routers use
> different route distinguisher (rd).
> Hence the pmbgpd does not receive a unique route corresponding to best
> path selected by the route-reflector, but the two or three different
> vpnv4 addresses (rd:a.b.c.d) corresponding to ip_prefix = a.b.c.d ?
>
> Wilfrid
>
>
> -Original Message-
> From: Grassot, Wilfrid 
> Sent: Monday, 18 May 2020 17:05
> To: Paolo Lucente ; pmacct-discussion@pmacct.net
> Subject: RE: [pmacct-discussion] BGP correlation not working with
> nfacctd, all BGP set to 0
>
> Hi Paolo,
>
> Thank you for your answer.
>
> My bad in the description of the issue:
> w.x.y.z is indeed the ipv4 address of the router loop0 which is also
> its router-id.
>
> Currently our setup is to iBGP peer with the router (router-id
> w.x.y.z) at the address-family vpnv4.
> We already filter out using route-target on the router for nfacctd  to
> receive only ipv4 routes from the monitored L3VPN.
> So the BGP daemon is only collecting routes of the monitored L3VPN
>
> On nfacctd collector we also receive only the netflow from routers
> interfaces configured on this vrf.
> If I manually make the correlation of the captured netflow, I can see
> in the BGP dump files the corresponding src_as, dest_as, peer_dst_ip
>
> So netflow and BGP are fine and bgp_agent_map file is  bgp_ip=w.x.y.z.
> ip=0.0.0.0/0 where w.x.y.z is the loopback0 (router-id) of the
router,
> and nfacctd is peering with it (sorry again for the mishap).
>
> I use the latest pmacctd 1.7.4 and I compile with ./configure
> --enable-jansson  (--enable-threads is not available)
>
> And yes our network is a confederation of 6 sub_as.
>
> Thank you
>
> Wilfrid Grassot
>
>
>
>
>
>
> -Original Message-
> From: Paolo Lucente 
> Sent: Monday, 18 May 2020 16:30
> To: pmacct-discussion@pmacct.net; Grassot, Wilfrid
> 
> Subject: Re: [pmacct-discussion] BGP correlation not working with
> nfacctd, all BGP set to 0
>
>
> Hi Wilfrid,
>
> Thanks for getting in touch. A couple of notes:
>
> 1) if you are sen