[pmacct-discussion] Ability to filter by as_src, net_src in tee plugin
Hi Paolo and the list, I have a necessity to filter out traffic data (based on src_as or src_net) to an IPFIX collector, capturing everything else. I was thinking that nfacctd in tee mode with pre_tag_map would be able to do that. However, according to pretag.map.example, the filtering is restricted while pmacct (nfacctd) is running in 'tee' mode: nfacctd when in 'tee' mode: valid keys: set_tag, set_tag2, set_label, ip, engine_type, engine_id, source_id Are there any plans for implementing pre_tag mapping/filtering based on packet data (ips, nets, asns etc)? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Sfacctd per-ASN + BGP traffic accounting
Hello, Paolo, Is it possible that is traffic hitting your own prefixes, It hardly could be for two reasons: - sflow is enabled only on "uplink" interfaces of the switch, facing to carriers. Traffic to my own prefixes wouldn't go there. - The BGP-to-pmacct export policy is set up to export internal routes. You may double-check by adding net_dst to your aggregation method and see for which prefixes you get a as_dst zero - and evaluate whether that makes sense or not. I did so, the as_dst=0 entries are represented with 0.0.0.0 nets (as I have suspected): MariaDB [pmacct]> select iface_out, as_dst, net_dst, packets, bytes from as_out order by stamp_inserted desc limit 20; +---++---+---+--+ | iface_out | as_dst | net_dst | packets | bytes| +---++---+---+--+ | 567 | 39572 | 46.229.160.0 | 4194304 | 6383730688 | | 564 | 12876 | 51.15.0.0 | 1048576 | 85983232 | | 567 | 35004 | 195.74.72.0 | 2097152 |134217728 | | 508 | 0 | 0.0.0.0 | 471859200 | 522110107648 | | 509 | 50113 | 185.180.228.0 |524288 | 38797312 | | 569 | 3269 | 79.0.0.0 | 2621440 | 3968860160 | | 564 | 15169 | 74.125.0.0| 30408704 | 2760900608 | | 564 | 60781 | 5.79.64.0 | 3145728 | 2136997888 | | 567 | 52000 | 185.15.211.0 | 8388608 | 12767461376 | | 509 | 12764 | 212.112.96.0 |524288 |797966336 | | 564 | 36873 | 105.112.22.0 | 1048576 | 1533018112 | | 567 | 49476 | 80.75.128.0 | 2097152 |465567744 | | 569 | 4837 | 112.192.0.0 |524288 |787480576 | | 508 | 35362 | 95.158.32.0 | 1048576 |335544320 | | 569 | 4804 | 49.187.32.0 |524288 |793772032 | | 564 | 37440 | 41.78.57.0| 1048576 | 1512046592 | | 564 | 54994 | 8.37.230.0| 1048576 | 77594624 | | 509 | 0 | 0.0.0.0 | 244318208 | 221750231040 | | 567 | 35816 | 78.30.192.0 | 4194304 |547356672 | | 567 | 6640 | 65.151.188.0 | 2097152 | 3191865344 | +---++---+---+--+ You could say that the routing table I have announced to sfacctd is incomplete but I announce all the prefixes my routers have got: they route traffic to uplinks according to it.. I guess pmacct's BGP daemon ignores or "loses" some announces. The routers sending BGP feed are Juniper MX480 and MX80. I'd be glad to help you with finding this bug, if you give me instructions what to check next. Paolo Lucente писал 2017-04-01 16:24: Hi Stanislaw, Is it possible that is traffic hitting your own prefixes, ie. prefixes lying in the same ASN you are iBGP peering with pmacct (for which the AS_PATH would be null and hence the as_dst would be null too)? You may double-check by adding net_dst to your aggregation method and see for which prefixes you get a as_dst zero - and evaluate whether that makes sense or not. Or, if you switch to GitHub master code (about to be freezed and rolled out in April as 1.6.2), you have a new bgp_daemon_as directive to define an ASN and hence allow you to set an eBGP peering with your router(s): this way you would see your own prefixes lying in your ASN instead of zeroes. Paolo On Thu, Mar 30, 2017 at 12:10:42PM +0300, Stanislaw wrote: Hello, I'm trying to setup sfacctd to account per-ASN traffic statistics using pmacct 1.6.1 (the latest). It seems to work but the topmost position in the report is dst_asn 0. Is it possible that pmacct can't handle BGP fullview, so portion of ASN<->prefix data is lost within the daemon? I use pmacct mysql table version 6 with iface_in, iface_out fields added. MariaDB [pmacct]> SELECT iface_out, as_dst, SUM(bytes) AS bytes FROM as_out GROUP BY iface_out, as_dst ORDER BY SUM(bytes) DESC LIMIT 20; +---+++ | iface_out | as_dst | bytes | +---+++ | 508 | 0 | 14271156862976 | | 509 | 0 | 8610211954688 | | 570 | 6849 | 6350382530560 | | 570 | 25229 | 6203280506880 | | 570 | 15169 | 3872144621568 | | 570 | 13188 | 3619018899456 | | 569 | 4134 | 3502273396736 | | 567 | 24940 | 2158440415232 | | 570 | 21343 | 2101956657152 | | 570 | 39608 | 2061838761984 | | 570 | 0 | 2005833433088 | | 570 | 13238 | 1953913880576 | | 570 | 21219 | 1912485134336 | | 569 | 3269 | 1716940570624 | | 570 | 31148 | 1612703350784 | | 570 | 6876 | 1525952118784 | | 564 | 15169 | 1316007411712 | | 570 | 3255 | 1223870398464 | | 567 | 133774 | 984798167040 | | 569 | 4837 | 978273566720 | +---+++ As far as I have got AS0 is an &quo
[pmacct-discussion] sfacctd: IPv6 per-ip or per-prefix accounting
Hi everybody, I've got a necessity to account traffic to/from my IPs within some prefixes (like, per-server traffic). With IPv4 I use the following configuration: plugins: mysql[traffic_in], mysql[traffic_out] aggregate[traffic_in]: src_net, dst_host aggregate[traffic_out]: src_host, dst_net aggregate_filter[traffic_in]: vlan and (dst net or dst net or dst net ) aggregate_filter[traffic_out]: vlan and (src net or src net or src net ) That config makes me able to get the traffic per-the-ip from the "everything to everything" sFlow got from the core switch. That works like a charm. Apparently, another approach to the IPv6 is needed as the similar aggregate_filter configuration doesn't work. I've tried: aggregate_filter[traffic_out_v6]: vlan and src net 2a00:1234::/32 Does anyone have some working IPv6 aggregate_filter examples? Also, is there a way to aggregate the IPv6 accounting to the /64 prefix basis, not to the each IP within my v6 space? Thanks for helping in advance! ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] sfacctd counts much less traffic than SNMP port stat
Hello, list. I'm using sfacctd to account traffic between autonomous systems at Internet Exchange Point (IXP). IXP runs on Extreme Networks x670 and BlackDiamond x8 switches with enabled sFlow accounting only on customer ports ingress traffic. I've noticed that sfacctd count much lesser traffic than SNMP port statistics. To compare sflow AS traffic and port statistics I've chosen one customer which announces only one AS, so theoretically AS traffic and port traffic must be almost identical. I looked into port graph with 1-hour interval: during hour bandwidth is raised from 10G to 12G (period from 14:00 to 15:00 on the attached graph) I ran this SQL query: select SUM(bytes) from flow_daily where as_src = 59613 and stamp_inserted = 1381402800; +--+ | SUM(bytes) | +--+ | 677594784256 | +--+ Where as_src is AS of that customer who announces only one AS, stamp_inserted is interval I measure. So this shows summary transmitted by that AS traffic during hour from 14:00 to 15:00. Converting transmitted bytes to average speed: 677594784256 * 8 = 5420758274048 bits 5420758274048 / 3600 = 1505766187 average bits per second 1505766187 / 1024 / 1024 / 1024 = 1.4Gbit per second. Details of my pmacct system: pmacct version 0.14.3 sfacctd config: syslog: local6 daemonize: true sfacctd_port: 6343 sfacctd_renormalize: true sfacctd_net: bgp sfacctd_as_new: bgp ! BGP settings bgp_daemon: true bgp_daemon_msglog: false bgp_peer_src_as_type: bgp bgp_daemon_ip: 91.245.221.251 bgp_daemon_port: 179 bgp_agent_map: /srv/pmacct/etc/bgp_agent_map sfacctd_ip: 172.17.172.3 plugins: mysql[daily], mysql[monthly], mysql[tmp] plugin_pipe_size[daily]: 68812800 plugin_pipe_size[monthly]: 68812800 plugin_pipe_size[tmp]: 68812800 plugin_buffer_size[daily]: 22400 plugin_buffer_size[monthly]: 22400 plugin_buffer_size[tmp]: 22400 aggregate[daily]: src_as, dst_as aggregate[monthly]: src_as, dst_as aggregate[tmp]: src_as, dst_as sql_table[daily]: flow_daily sql_table[monthly]: flow_monthly sql_table[tmp]: flow_tmp sql_host[daily]: localhost sql_host[monthly]: localhost sql_host[tmp]: localhost sql_user[daily]: pmacct sql_user[monthly]: pmacct sql_user[tmp]: pmacct sql_passwd[daily]: 1 sql_passwd[monthly]: 1 sql_passwd[tmp]: 1 sql_db[daily]: pmacct sql_db[monthly]: pmacct sql_db[tmp]: pmacct sql_table_version[daily]: 6 sql_table_version[monthly]: 6 sql_table_version[tmp]: 6 sql_dont_try_update[daily]: true sql_dont_try_update[monthly]: true sql_dont_try_update[tmp]: true sql_multi_values[daily]: 14000 sql_multi_values[monthly]: 14000 sql_multi_values[tmp]: 14000 sql_cache_entries[daily]: 1403 sql_cache_entries[monthly]: 1403 sql_cache_entries[tmp]: 1403 sql_locking_style[daily]: row sql_locking_style[monthly]: row sql_locking_style[tmp]: row sfacctd_disable_checks: true sql_history_since_epoch[daily]: true sql_history_since_epoch[monthly]: true sql_history_since_epoch[tmp]: true sql_history_roundoff[daily]: h sql_history_roundoff[monthly]: d sql_history_roundoff[tmp]: h sql_history[daily]: 1h sql_history[monthly]: 1d sql_history[tmp]: 1h sql_refresh_time[daily]: 3600 sql_refresh_time[monthly]: 86400 sql_refresh_time[tmp]: 3600 My MySQL table structure: | flow_daily | CREATE TABLE `flow_daily` ( `agent_id` int(4) unsigned NOT NULL, `class_id` char(16) NOT NULL, `mac_src` char(17) NOT NULL, `mac_dst` char(17) NOT NULL, `vlan` int(2) unsigned NOT NULL, `as_src` int(4) unsigned NOT NULL, `as_dst` int(4) unsigned NOT NULL, `ip_src` char(15) NOT NULL, `ip_dst` char(15) NOT NULL, `src_port` int(2) unsigned NOT NULL, `dst_port` int(2) unsigned NOT NULL, `ip_proto` char(6) NOT NULL, `tos` int(4) unsigned NOT NULL, `packets` int(10) unsigned NOT NULL, `bytes` bigint(20) unsigned NOT NULL, `flows` int(10) unsigned NOT NULL, `stamp_inserted` int(8) NOT NULL DEFAULT '0', `stamp_updated` int(8) DEFAULT NULL, PRIMARY KEY (`stamp_inserted`,`as_src`,`as_dst`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 /*!50100 PARTITION BY RANGE (stamp_inserted) (PARTITION p2013_09_08 VALUES LESS THAN (1378674000) ENGINE = InnoDB, PARTITION p2013_09_09 VALUES LESS THAN (1378760400) ENGINE = InnoDB, PARTITION p2013_09_10 VALUES LESS THAN (1378846800) ENGINE = InnoDB, PARTITION p2013_09_11 VALUES LESS THAN (1378933200) ENGINE = InnoDB, PARTITION p2013_09_12 VALUES LESS THAN (1379019600) ENGINE = InnoDB, PARTITION p2013_09_13 VALUES LESS THAN (1379106000) ENGINE = InnoDB, PARTITION p2013_09_14 VALUES LESS THAN (1379192400) ENGINE = InnoDB, PARTITION p2013_09_15 VALUES LESS THAN (1379278800) ENGINE = InnoDB, PARTITION p2013_09_16 VALUES LESS THAN (1379365200) ENGINE = InnoDB, PARTITION p2013_09_17 VALUES LESS THAN (1379451600) ENGINE = InnoDB, PARTITION p2013_09_18 VALUES LESS THAN (1379538000) ENGINE = InnoDB, PARTITION p2013_09_19 VALUES LESS THAN (1379624400) ENGINE = InnoDB, PARTITION p2013_09_20 VALUES LESS THAN (1379710800) ENGINE =
