Re: [pmacct-discussion] pmacct 1.7.1 released !
Keep up the good work Paolo and thanx for this excellent software! Alex On Sun, May 6, 2018 at 4:44 PM, Paolo Lucente wrote: > VERSION. > 1.7.1 > > > DESCRIPTION. > pmacct is a small set of multi-purpose passive network monitoring tools. It > can account, classify, aggregate, replicate and export forwarding-plane > data, > ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP > and BMP; collect infrastructure data via Streaming Telemetry. Each > component > works both as a standalone daemon and as a thread of execution for > correlation > purposes (ie. enrich NetFlow with BGP data). > > A pluggable architecture allows to store collected forwarding-plane data > into > memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, > BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > pmacct offers customizable historical data breakdown, data enrichments like > BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. > Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX > are > all supported as inputs for forwarding-plane data. Replication of incoming > NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be > easily exported to time-series databases like ElasticSearch and InfluxDB > and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. > > Control-plane and infrastructure data, collected via BGP, BMP and Streaming > Telemetry, can be all logged real-time or dumped at regular time intervals > to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > > > HOMEPAGE. > http://www.pmacct.net/ > > > DOWNLOAD. > http://www.pmacct.net/pmacct-1.7.1.tar.gz > > > CHANGELOG. > + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers > (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a > standalone BGP daemon (pmbgpd). The aim is to facilitate operations > when re-sizing/re-balancing the collection infrastructure without > impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map > expects full pathname to a file where cross-connects are defined; > mapping works only against the IP source address and not the BGP > Router ID, only 1:1 relationships can be formed (ie. this is about > cross-connecting, not replication) and only one session per BGP > peer is supported (ie. multiple BGP agents are running on the same > IP address or NAT traversal scenarios are not supported [yet]). > A sample map is provided in 'examples/bgp_xconnects.map.example'. > + pmbgpd: introduced a BGP Looking Glass server allowing to perform > queries, ie. lookup of IP addresses/prefixes or get the list of BGP > peers, against available BGP RIBs. The server is asyncronous and > uses ZeroMQ as transport layer to serve incoming queries. Sample > C/Python LG clients are available in 'examples/lg'. A sample LG > server config is available in QUICKSTART. Request/Reply Looking > Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'. > + pmacctd: a single daemon can now listen for traffic on multiple > interfaces via a polling mechanism. This can be configured via a > pcap_interfaces_map feature (interface/pcap_interface can still be > used for backward compatiblity to listen on a single interface). The > map allows to define also ifindex mapping and capturing direction on > a per-interface basis. The map can be reloaded at runtime via a USR2 > signal and a sample map is in examples/pcap_interfaces.map.example. > + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and > kafka_partition_key knobs is introduced. The Kafka topic can contain > variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which > are all computed when data is purged to the backend. This feature is > in addition to the existing kafka_partition feature which allows to > rely on the built-in Kafka partitioning to assign data statically to > one partition or rely dynamically on the default partitioner. The > feature is courtesy by Corentin Neau / Codethink ( @weyfonk ). > + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone > represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives > the timestamps_rfc3339 knob can be used to enable this feature (left > disabled by default for backward compatibility). > + timestamps_utc: new knob to decode timestamps to UTC timezone even > if the Operating System is set to a different timezone. On the goods > of running a system set to UTC please read Q18 of FAQS. > + sfacctd: implemented mpls_label_top, mpls_label_bottom and > mpls_stack_depth primitives decoded from sFlow flow sample headers. > Thanks to David Barroso ( @dbarrosop ) for his support. > + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131 > (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX > option packets (these IEs were already supported when passed in flow > data)
Re: [pmacct-discussion] pmacct 1.7.1 released !
Congratulations, Paolo, these are really great updates! Cheers, and thanks again for all of your hard work for the community. Aaron On Sun, May 6, 2018, 6:45 AM Paolo Lucente wrote: > VERSION. > 1.7.1 > > > DESCRIPTION. > pmacct is a small set of multi-purpose passive network monitoring tools. It > can account, classify, aggregate, replicate and export forwarding-plane > data, > ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP > and BMP; collect infrastructure data via Streaming Telemetry. Each > component > works both as a standalone daemon and as a thread of execution for > correlation > purposes (ie. enrich NetFlow with BGP data). > > A pluggable architecture allows to store collected forwarding-plane data > into > memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, > BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > pmacct offers customizable historical data breakdown, data enrichments like > BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. > Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX > are > all supported as inputs for forwarding-plane data. Replication of incoming > NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be > easily exported to time-series databases like ElasticSearch and InfluxDB > and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. > > Control-plane and infrastructure data, collected via BGP, BMP and Streaming > Telemetry, can be all logged real-time or dumped at regular time intervals > to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > > > HOMEPAGE. > http://www.pmacct.net/ > > > DOWNLOAD. > http://www.pmacct.net/pmacct-1.7.1.tar.gz > > > CHANGELOG. > + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers > (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a > standalone BGP daemon (pmbgpd). The aim is to facilitate operations > when re-sizing/re-balancing the collection infrastructure without > impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map > expects full pathname to a file where cross-connects are defined; > mapping works only against the IP source address and not the BGP > Router ID, only 1:1 relationships can be formed (ie. this is about > cross-connecting, not replication) and only one session per BGP > peer is supported (ie. multiple BGP agents are running on the same > IP address or NAT traversal scenarios are not supported [yet]). > A sample map is provided in 'examples/bgp_xconnects.map.example'. > + pmbgpd: introduced a BGP Looking Glass server allowing to perform > queries, ie. lookup of IP addresses/prefixes or get the list of BGP > peers, against available BGP RIBs. The server is asyncronous and > uses ZeroMQ as transport layer to serve incoming queries. Sample > C/Python LG clients are available in 'examples/lg'. A sample LG > server config is available in QUICKSTART. Request/Reply Looking > Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'. > + pmacctd: a single daemon can now listen for traffic on multiple > interfaces via a polling mechanism. This can be configured via a > pcap_interfaces_map feature (interface/pcap_interface can still be > used for backward compatiblity to listen on a single interface). The > map allows to define also ifindex mapping and capturing direction on > a per-interface basis. The map can be reloaded at runtime via a USR2 > signal and a sample map is in examples/pcap_interfaces.map.example. > + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and > kafka_partition_key knobs is introduced. The Kafka topic can contain > variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which > are all computed when data is purged to the backend. This feature is > in addition to the existing kafka_partition feature which allows to > rely on the built-in Kafka partitioning to assign data statically to > one partition or rely dynamically on the default partitioner. The > feature is courtesy by Corentin Neau / Codethink ( @weyfonk ). > + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone > represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives > the timestamps_rfc3339 knob can be used to enable this feature (left > disabled by default for backward compatibility). > + timestamps_utc: new knob to decode timestamps to UTC timezone even > if the Operating System is set to a different timezone. On the goods > of running a system set to UTC please read Q18 of FAQS. > + sfacctd: implemented mpls_label_top, mpls_label_bottom and > mpls_stack_depth primitives decoded from sFlow flow sample headers. > Thanks to David Barroso ( @dbarrosop ) for his support. > + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131 > (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX > option packets (these