Re: [pmacct-discussion] Sfacctd per-ASN + BGP traffic accounting

2017-04-06 Thread Paolo Lucente 

Hi Stanislaw,

Interesting. Can you please modify:

sfacctd_net: bgp
sfacctd_as_new: bgp

Into:

sfacctd_net: fallback
sfacctd_as_new: fallback

This should prevent net_dst to be zero, except for some reason that is
actually comes as zero in the flow itself. Send me some output of that;
i propose let's continue by unicast email and summarise results here on
list.

Paolo

On Thu, Apr 06, 2017 at 03:02:20PM +0300, Stanislaw wrote:
> Hello, Paolo,
> 
> >Is it possible that is traffic hitting your own prefixes,
> It hardly could be for two reasons:
> - sflow is enabled only on "uplink" interfaces of the switch, facing
> to carriers. Traffic to my own prefixes wouldn't go there.
> - The BGP-to-pmacct export policy is set up to export internal routes.
> 
> >You may double-check by adding net_dst to your aggregation method
> >and see for which prefixes you get a as_dst zero - and evaluate
> >whether that makes sense or not.
> I did so, the as_dst=0 entries are represented with 0.0.0.0 nets (as
> I have suspected):
> MariaDB [pmacct]> select iface_out, as_dst, net_dst, packets, bytes
> from as_out order by stamp_inserted desc limit 20;
> +---++---+---+--+
> | iface_out | as_dst | net_dst   | packets   | bytes|
> +---++---+---+--+
> |   567 |  39572 | 46.229.160.0  |   4194304 |   6383730688 |
> |   564 |  12876 | 51.15.0.0 |   1048576 | 85983232 |
> |   567 |  35004 | 195.74.72.0   |   2097152 |134217728 |
> |   508 |  0 | 0.0.0.0   | 471859200 | 522110107648 |
> |   509 |  50113 | 185.180.228.0 |524288 | 38797312 |
> |   569 |   3269 | 79.0.0.0  |   2621440 |   3968860160 |
> |   564 |  15169 | 74.125.0.0|  30408704 |   2760900608 |
> |   564 |  60781 | 5.79.64.0 |   3145728 |   2136997888 |
> |   567 |  52000 | 185.15.211.0  |   8388608 |  12767461376 |
> |   509 |  12764 | 212.112.96.0  |524288 |797966336 |
> |   564 |  36873 | 105.112.22.0  |   1048576 |   1533018112 |
> |   567 |  49476 | 80.75.128.0   |   2097152 |465567744 |
> |   569 |   4837 | 112.192.0.0   |524288 |787480576 |
> |   508 |  35362 | 95.158.32.0   |   1048576 |335544320 |
> |   569 |   4804 | 49.187.32.0   |524288 |793772032 |
> |   564 |  37440 | 41.78.57.0|   1048576 |   1512046592 |
> |   564 |  54994 | 8.37.230.0|   1048576 | 77594624 |
> |   509 |  0 | 0.0.0.0   | 244318208 | 221750231040 |
> |   567 |  35816 | 78.30.192.0   |   4194304 |547356672 |
> |   567 |   6640 | 65.151.188.0  |   2097152 |   3191865344 |
> +---++---+---+--+
> 
> 
> You could say that the routing table I have announced to sfacctd is
> incomplete but I announce all the prefixes my routers have got: they
> route traffic to uplinks according to it.. I guess pmacct's BGP
> daemon ignores or "loses" some announces. The routers sending BGP
> feed are Juniper MX480 and MX80.
> 
> I'd be glad to help you with finding this bug, if you give me
> instructions what to check next.
> 
> 
> 
> Paolo Lucente писал 2017-04-01 16:24:
> >Hi Stanislaw,
> >
> >Is it possible that is traffic hitting your own prefixes, ie. prefixes
> >lying in the same ASN you are iBGP peering with pmacct (for which the
> >AS_PATH would be null and hence the as_dst would be null too)? You may
> >double-check by adding net_dst to your aggregation method and see for
> >which prefixes you get a as_dst zero - and evaluate whether that makes
> >sense or not. Or, if you switch to GitHub master code (about to be
> >freezed and rolled out in April as 1.6.2), you have a new bgp_daemon_as
> >directive to define an ASN and hence allow you to set an eBGP peering
> >with your router(s): this way you would see your own prefixes lying in
> >your ASN instead of zeroes.
> >
> >Paolo
> >
> >On Thu, Mar 30, 2017 at 12:10:42PM +0300, Stanislaw wrote:
> >>Hello,
> >>I'm trying to setup sfacctd to account per-ASN traffic statistics
> >>using pmacct 1.6.1 (the latest). It seems to work but the topmost
> >>position in the report is dst_asn 0.
> >>Is it possible that pmacct can't handle BGP fullview, so portion of
> >>ASN<->prefix data is lost within the daemon?
> >>
> >>
> >>I use pmacct mysql table version 6 with iface_in, iface_out fields
> >>added.
> >>MariaDB [pmacct]> SELECT iface_out, as_dst, SUM(bytes) AS bytes FROM
> >>as_out GROUP BY iface_out, as_dst ORDER BY SUM(bytes) DESC LIMIT 20;
> >>+---+++
> >>| iface_out | as_dst | bytes  |
> >>+---+++
> >>|   508 |  0 | 14271156862976 |
> >>|   509 |  0 |  8610211954688 |
> >>|   570 |   6849 |  6350382530560 |
> >>|   570 |  25229 |  6203280506880 |
> >>|   570 |  15169 |  3872144621568 |
> >>|   570 |  13188 |  3619018899456 |
> >>|

Re: [pmacct-discussion] Sfacctd per-ASN + BGP traffic accounting

2017-04-06 Thread Stanislaw 

Hello, Paolo,


Is it possible that is traffic hitting your own prefixes,

It hardly could be for two reasons:
- sflow is enabled only on "uplink" interfaces of the switch, facing to 
carriers. Traffic to my own prefixes wouldn't go there.

- The BGP-to-pmacct export policy is set up to export internal routes.

You may double-check by adding net_dst to your aggregation method and 
see for which prefixes you get a as_dst zero - and evaluate whether 
that makes sense or not.
I did so, the as_dst=0 entries are represented with 0.0.0.0 nets (as I 
have suspected):
MariaDB [pmacct]> select iface_out, as_dst, net_dst, packets, bytes from 
as_out order by stamp_inserted desc limit 20;

+---++---+---+--+
| iface_out | as_dst | net_dst   | packets   | bytes|
+---++---+---+--+
|   567 |  39572 | 46.229.160.0  |   4194304 |   6383730688 |
|   564 |  12876 | 51.15.0.0 |   1048576 | 85983232 |
|   567 |  35004 | 195.74.72.0   |   2097152 |134217728 |
|   508 |  0 | 0.0.0.0   | 471859200 | 522110107648 |
|   509 |  50113 | 185.180.228.0 |524288 | 38797312 |
|   569 |   3269 | 79.0.0.0  |   2621440 |   3968860160 |
|   564 |  15169 | 74.125.0.0|  30408704 |   2760900608 |
|   564 |  60781 | 5.79.64.0 |   3145728 |   2136997888 |
|   567 |  52000 | 185.15.211.0  |   8388608 |  12767461376 |
|   509 |  12764 | 212.112.96.0  |524288 |797966336 |
|   564 |  36873 | 105.112.22.0  |   1048576 |   1533018112 |
|   567 |  49476 | 80.75.128.0   |   2097152 |465567744 |
|   569 |   4837 | 112.192.0.0   |524288 |787480576 |
|   508 |  35362 | 95.158.32.0   |   1048576 |335544320 |
|   569 |   4804 | 49.187.32.0   |524288 |793772032 |
|   564 |  37440 | 41.78.57.0|   1048576 |   1512046592 |
|   564 |  54994 | 8.37.230.0|   1048576 | 77594624 |
|   509 |  0 | 0.0.0.0   | 244318208 | 221750231040 |
|   567 |  35816 | 78.30.192.0   |   4194304 |547356672 |
|   567 |   6640 | 65.151.188.0  |   2097152 |   3191865344 |
+---++---+---+--+


You could say that the routing table I have announced to sfacctd is 
incomplete but I announce all the prefixes my routers have got: they 
route traffic to uplinks according to it.. I guess pmacct's BGP daemon 
ignores or "loses" some announces. The routers sending BGP feed are 
Juniper MX480 and MX80.


I'd be glad to help you with finding this bug, if you give me 
instructions what to check next.




Paolo Lucente писал 2017-04-01 16:24:

Hi Stanislaw,

Is it possible that is traffic hitting your own prefixes, ie. prefixes
lying in the same ASN you are iBGP peering with pmacct (for which the
AS_PATH would be null and hence the as_dst would be null too)? You may
double-check by adding net_dst to your aggregation method and see for
which prefixes you get a as_dst zero - and evaluate whether that makes
sense or not. Or, if you switch to GitHub master code (about to be
freezed and rolled out in April as 1.6.2), you have a new bgp_daemon_as
directive to define an ASN and hence allow you to set an eBGP peering
with your router(s): this way you would see your own prefixes lying in
your ASN instead of zeroes.

Paolo

On Thu, Mar 30, 2017 at 12:10:42PM +0300, Stanislaw wrote:

Hello,
I'm trying to setup sfacctd to account per-ASN traffic statistics
using pmacct 1.6.1 (the latest). It seems to work but the topmost
position in the report is dst_asn 0.
Is it possible that pmacct can't handle BGP fullview, so portion of
ASN<->prefix data is lost within the daemon?


I use pmacct mysql table version 6 with iface_in, iface_out fields
added.
MariaDB [pmacct]> SELECT iface_out, as_dst, SUM(bytes) AS bytes FROM
as_out GROUP BY iface_out, as_dst ORDER BY SUM(bytes) DESC LIMIT 20;
+---+++
| iface_out | as_dst | bytes  |
+---+++
|   508 |  0 | 14271156862976 |
|   509 |  0 |  8610211954688 |
|   570 |   6849 |  6350382530560 |
|   570 |  25229 |  6203280506880 |
|   570 |  15169 |  3872144621568 |
|   570 |  13188 |  3619018899456 |
|   569 |   4134 |  3502273396736 |
|   567 |  24940 |  2158440415232 |
|   570 |  21343 |  2101956657152 |
|   570 |  39608 |  2061838761984 |
|   570 |  0 |  2005833433088 |
|   570 |  13238 |  1953913880576 |
|   570 |  21219 |  1912485134336 |
|   569 |   3269 |  1716940570624 |
|   570 |  31148 |  1612703350784 |
|   570 |   6876 |  1525952118784 |
|   564 |  15169 |  1316007411712 |
|   570 |   3255 |  1223870398464 |
|   567 | 133774 |   984798167040 |
|   569 |   4837 |   978273566720 |
+---+++

As far as I have got AS0 is an "ASN which pmacct doesn't know about"

Re: [pmacct-discussion] sfacctd and tagging with filters

2017-04-06 Thread Aurelien Kempiak 

Yes, it's VLAN-tagged. I've tested your suggestion, it works !!

Was I able to find this info myself (I can't find it in the 
documentation), or was it quite difficult to find it without asking here ?


I'm asking this in order to improve my autonomy. Thanks a lot Paolo !




Le 06/04/2017 à 10:47, Paolo Lucente a écrit :

Hi Aurelien,

Is it possible your traffic is VLAN-tagged and/or MPLS-labelled? That
may explain why tagging is not working, pcap filters are sensible to
that. See for example here:

https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg02784.html

Paolo
  
On Tue, Apr 04, 2017 at 04:32:13PM +0200, Aurelien Kempiak wrote:

Hello there!

I'm trying to tag my flows per set of hosts (each set is a customer)
and record them into mysql.
So, here is my (testing) pretag.map file :

set_tag=101 ip=185.151.188.33 filter='src host 185.151.188.30'
set_tag=102 ip=185.151.188.34 filter='src host 185.151.188.30'
set_tag=103 ip=0.0.0.0/0 filter='src net 185.151.188.0/22'
set_tag=106 ip=0.0.0.0/0

And the relevant part of my sfacctd.conf file :

! tag filtering
pre_tag_map: /etc/pmacct/pretag.map
pre_tag_filter[if-out]: 101-106

! aggregate definitions
aggregate[if-out]: tag, src_host

With that configuration, flows done by 185.151.188.30 are well
recorded with '185.151.188.30' value in the 'ip_src' mysql field,
but they are never tagged with '101', nor '102', nor even '103'
values. They are tagged with '106' value. Like if the filter never
matches.
I have no idea why, could someone help please ? Thanks a lot !


--



*Aurélien* *Kempiak*
*System & Network Engineer*

*Fixe :* 03 59 82 20 05

125 Avenue de la République 59110 La Madeleine
12 rue Marivaux 75002 Paris






___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


--



*Aurélien* *Kempiak*
*System & Network Engineer*

*Fixe :* 03 59 82 20 05

125 Avenue de la République 59110 La Madeleine
12 rue Marivaux 75002 Paris

 
 
 



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sfacctd and tagging with filters

2017-04-06 Thread Paolo Lucente 

Hi Aurelien,

Is it possible your traffic is VLAN-tagged and/or MPLS-labelled? That
may explain why tagging is not working, pcap filters are sensible to
that. See for example here:

https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg02784.html

Paolo
 
On Tue, Apr 04, 2017 at 04:32:13PM +0200, Aurelien Kempiak wrote:
> Hello there!
> 
> I'm trying to tag my flows per set of hosts (each set is a customer)
> and record them into mysql.
> So, here is my (testing) pretag.map file :
> 
> set_tag=101 ip=185.151.188.33 filter='src host 185.151.188.30'
> set_tag=102 ip=185.151.188.34 filter='src host 185.151.188.30'
> set_tag=103 ip=0.0.0.0/0 filter='src net 185.151.188.0/22'
> set_tag=106 ip=0.0.0.0/0
> 
> And the relevant part of my sfacctd.conf file :
> 
> ! tag filtering
> pre_tag_map: /etc/pmacct/pretag.map
> pre_tag_filter[if-out]: 101-106
> 
> ! aggregate definitions
> aggregate[if-out]: tag, src_host
> 
> With that configuration, flows done by 185.151.188.30 are well
> recorded with '185.151.188.30' value in the 'ip_src' mysql field,
> but they are never tagged with '101', nor '102', nor even '103'
> values. They are tagged with '106' value. Like if the filter never
> matches.
> I have no idea why, could someone help please ? Thanks a lot !
> 
> 
> -- 
> 
>   
> 
> *Aurélien* *Kempiak*
> *System & Network Engineer*
> 
> *Fixe :* 03 59 82 20 05
> 
> 125 Avenue de la République 59110 La Madeleine
> 12 rue Marivaux 75002 Paris
> 
> 
> 
> 
> 
> 

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists