Re: [pmacct-discussion] pmacct 1.7.1 released !

[Alex K]

Keep up the good work Paolo and thanx for this excellent software!

Alex

On Sun, May 6, 2018 at 4:44 PM, Paolo Lucente  wrote:

> VERSION.
> 1.7.1
>
>
> DESCRIPTION.
> pmacct is a small set of multi-purpose passive network monitoring tools. It
> can account, classify, aggregate, replicate and export forwarding-plane
> data,
> ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
> and BMP; collect infrastructure data via Streaming Telemetry. Each
> component
> works both as a standalone daemon and as a thread of execution for
> correlation
> purposes (ie. enrich NetFlow with BGP data).
>
> A pluggable architecture allows to store collected forwarding-plane data
> into
> memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
> BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
> pmacct offers customizable historical data breakdown, data enrichments like
> BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
> Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX
> are
> all supported as inputs for forwarding-plane data. Replication of incoming
> NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
> easily exported to time-series databases like ElasticSearch and InfluxDB
> and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.
>
> Control-plane and infrastructure data, collected via BGP, BMP and Streaming
> Telemetry, can be all logged real-time or dumped at regular time intervals
> to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
>
>
> HOMEPAGE.
> http://www.pmacct.net/
>
>
> DOWNLOAD.
> http://www.pmacct.net/pmacct-1.7.1.tar.gz
>
>
> CHANGELOG.
> + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers
>   (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a
>   standalone BGP daemon (pmbgpd). The aim is to facilitate operations
>   when re-sizing/re-balancing the collection infrastructure without
>   impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map
>   expects full pathname to a file where cross-connects are defined;
>   mapping works only against the IP source address and not the BGP
>   Router ID, only 1:1 relationships can be formed (ie. this is about
>   cross-connecting, not replication) and only one session per BGP
>   peer is supported (ie. multiple BGP agents are running on the same
>   IP address or NAT traversal scenarios are not supported [yet]).
>   A sample map is provided in 'examples/bgp_xconnects.map.example'.
> + pmbgpd: introduced a BGP Looking Glass server allowing to perform
>   queries, ie. lookup of IP addresses/prefixes or get the list of BGP
>   peers, against available BGP RIBs. The server is asyncronous and
>   uses ZeroMQ as transport layer to serve incoming queries. Sample
>   C/Python LG clients are available in 'examples/lg'. A sample LG
>   server config is available in QUICKSTART. Request/Reply Looking
>   Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'.
> + pmacctd: a single daemon can now listen for traffic on multiple
>   interfaces via a polling mechanism. This can be configured via a
>   pcap_interfaces_map feature (interface/pcap_interface can still be
>   used for backward compatiblity to listen on a single interface). The
>   map allows to define also ifindex mapping and capturing direction on
>   a per-interface basis. The map can be reloaded at runtime via a USR2
>   signal and a sample map is in examples/pcap_interfaces.map.example.
> + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and
>   kafka_partition_key knobs is introduced. The Kafka topic can contain
>   variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which
>   are all computed when data is purged to the backend. This feature is
>   in addition to the existing kafka_partition feature which allows to
>   rely on the built-in Kafka partitioning to assign data statically to
>   one partition or rely dynamically on the default partitioner. The
>   feature is courtesy by Corentin Neau / Codethink ( @weyfonk ).
> + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone
>   represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives
>   the timestamps_rfc3339 knob can be used to enable this feature (left
>   disabled by default for backward compatibility).
> + timestamps_utc: new knob to decode timestamps to UTC timezone even
>   if the Operating System is set to a different timezone. On the goods
>   of running a system set to UTC please read Q18 of FAQS.
> + sfacctd: implemented mpls_label_top, mpls_label_bottom and
>   mpls_stack_depth primitives decoded from sFlow flow sample headers.
>   Thanks to David Barroso ( @dbarrosop ) for his support.
> + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131
>   (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX
>   option packets (these IEs were already supported when 

Re: [pmacct-discussion] pmacct 1.7.1 released !

[Aaron Finney]

Congratulations, Paolo, these are really great updates! Cheers, and thanks
again for all of your hard work for the community.

Aaron

On Sun, May 6, 2018, 6:45 AM Paolo Lucente  wrote:

> VERSION.
> 1.7.1
>
>
> DESCRIPTION.
> pmacct is a small set of multi-purpose passive network monitoring tools. It
> can account, classify, aggregate, replicate and export forwarding-plane
> data,
> ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
> and BMP; collect infrastructure data via Streaming Telemetry. Each
> component
> works both as a standalone daemon and as a thread of execution for
> correlation
> purposes (ie. enrich NetFlow with BGP data).
>
> A pluggable architecture allows to store collected forwarding-plane data
> into
> memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
> BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
> pmacct offers customizable historical data breakdown, data enrichments like
> BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
> Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX
> are
> all supported as inputs for forwarding-plane data. Replication of incoming
> NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
> easily exported to time-series databases like ElasticSearch and InfluxDB
> and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.
>
> Control-plane and infrastructure data, collected via BGP, BMP and Streaming
> Telemetry, can be all logged real-time or dumped at regular time intervals
> to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
>
>
> HOMEPAGE.
> http://www.pmacct.net/
>
>
> DOWNLOAD.
> http://www.pmacct.net/pmacct-1.7.1.tar.gz
>
>
> CHANGELOG.
> + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers
>   (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a
>   standalone BGP daemon (pmbgpd). The aim is to facilitate operations
>   when re-sizing/re-balancing the collection infrastructure without
>   impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map
>   expects full pathname to a file where cross-connects are defined;
>   mapping works only against the IP source address and not the BGP
>   Router ID, only 1:1 relationships can be formed (ie. this is about
>   cross-connecting, not replication) and only one session per BGP
>   peer is supported (ie. multiple BGP agents are running on the same
>   IP address or NAT traversal scenarios are not supported [yet]).
>   A sample map is provided in 'examples/bgp_xconnects.map.example'.
> + pmbgpd: introduced a BGP Looking Glass server allowing to perform
>   queries, ie. lookup of IP addresses/prefixes or get the list of BGP
>   peers, against available BGP RIBs. The server is asyncronous and
>   uses ZeroMQ as transport layer to serve incoming queries. Sample
>   C/Python LG clients are available in 'examples/lg'. A sample LG
>   server config is available in QUICKSTART. Request/Reply Looking
>   Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'.
> + pmacctd: a single daemon can now listen for traffic on multiple
>   interfaces via a polling mechanism. This can be configured via a
>   pcap_interfaces_map feature (interface/pcap_interface can still be
>   used for backward compatiblity to listen on a single interface). The
>   map allows to define also ifindex mapping and capturing direction on
>   a per-interface basis. The map can be reloaded at runtime via a USR2
>   signal and a sample map is in examples/pcap_interfaces.map.example.
> + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and
>   kafka_partition_key knobs is introduced. The Kafka topic can contain
>   variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which
>   are all computed when data is purged to the backend. This feature is
>   in addition to the existing kafka_partition feature which allows to
>   rely on the built-in Kafka partitioning to assign data statically to
>   one partition or rely dynamically on the default partitioner. The
>   feature is courtesy by Corentin Neau / Codethink ( @weyfonk ).
> + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone
>   represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives
>   the timestamps_rfc3339 knob can be used to enable this feature (left
>   disabled by default for backward compatibility).
> + timestamps_utc: new knob to decode timestamps to UTC timezone even
>   if the Operating System is set to a different timezone. On the goods
>   of running a system set to UTC please read Q18 of FAQS.
> + sfacctd: implemented mpls_label_top, mpls_label_bottom and
>   mpls_stack_depth primitives decoded from sFlow flow sample headers.
>   Thanks to David Barroso ( @dbarrosop ) for his support.
> + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131
>   (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX
>   

[pmacct-discussion] pmacct 1.7.1 released !

[Paolo Lucente]

VERSION.
1.7.1


DESCRIPTION.
pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry. Each component
works both as a standalone daemon and as a thread of execution for correlation
purposes (ie. enrich NetFlow with BGP data).

A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to time-series databases like ElasticSearch and InfluxDB
and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.


HOMEPAGE.
http://www.pmacct.net/


DOWNLOAD.
http://www.pmacct.net/pmacct-1.7.1.tar.gz


CHANGELOG.
+ pmbgpd: introduced a BGP x-connect feature meant to map BGP peers
  (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a
  standalone BGP daemon (pmbgpd). The aim is to facilitate operations
  when re-sizing/re-balancing the collection infrastructure without
  impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map
  expects full pathname to a file where cross-connects are defined;
  mapping works only against the IP source address and not the BGP
  Router ID, only 1:1 relationships can be formed (ie. this is about
  cross-connecting, not replication) and only one session per BGP
  peer is supported (ie. multiple BGP agents are running on the same
  IP address or NAT traversal scenarios are not supported [yet]).
  A sample map is provided in 'examples/bgp_xconnects.map.example'.
+ pmbgpd: introduced a BGP Looking Glass server allowing to perform
  queries, ie. lookup of IP addresses/prefixes or get the list of BGP
  peers, against available BGP RIBs. The server is asyncronous and
  uses ZeroMQ as transport layer to serve incoming queries. Sample
  C/Python LG clients are available in 'examples/lg'. A sample LG
  server config is available in QUICKSTART. Request/Reply Looking
  Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'.
+ pmacctd: a single daemon can now listen for traffic on multiple
  interfaces via a polling mechanism. This can be configured via a
  pcap_interfaces_map feature (interface/pcap_interface can still be
  used for backward compatiblity to listen on a single interface). The
  map allows to define also ifindex mapping and capturing direction on
  a per-interface basis. The map can be reloaded at runtime via a USR2
  signal and a sample map is in examples/pcap_interfaces.map.example.
+ Kafka plugin: dynamic partitioning via kafka_partition_dynamic and
  kafka_partition_key knobs is introduced. The Kafka topic can contain
  variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which
  are all computed when data is purged to the backend. This feature is
  in addition to the existing kafka_partition feature which allows to
  rely on the built-in Kafka partitioning to assign data statically to
  one partition or rely dynamically on the default partitioner. The
  feature is courtesy by Corentin Neau / Codethink ( @weyfonk ).
+ Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone
  represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives
  the timestamps_rfc3339 knob can be used to enable this feature (left
  disabled by default for backward compatibility).
+ timestamps_utc: new knob to decode timestamps to UTC timezone even
  if the Operating System is set to a different timezone. On the goods
  of running a system set to UTC please read Q18 of FAQS.
+ sfacctd: implemented mpls_label_top, mpls_label_bottom and
  mpls_stack_depth primitives decoded from sFlow flow sample headers.
  Thanks to David Barroso ( @dbarrosop ) for his support.
+ nfacctd: added support for IEs 130 (exporterIPv4Address) and 131
  (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX
  option packets (these IEs were already supported when passed in flow
  data). Also added support for IE 351 (dataLinkFrameSection) which
  carries the initial portion of a sampled raw packet headers (a-la
  sFlow). This was tested working against a Cisco NCS 5k platform.
+ nfprobe plugin: added a new nfprobe_dont_cache knob allowing to
  disable caching and summarisation