VERSION. 1.7.1
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.1.tar.gz CHANGELOG. + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a standalone BGP daemon (pmbgpd). The aim is to facilitate operations when re-sizing/re-balancing the collection infrastructure without impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map expects full pathname to a file where cross-connects are defined; mapping works only against the IP source address and not the BGP Router ID, only 1:1 relationships can be formed (ie. this is about cross-connecting, not replication) and only one session per BGP peer is supported (ie. multiple BGP agents are running on the same IP address or NAT traversal scenarios are not supported [yet]). A sample map is provided in 'examples/bgp_xconnects.map.example'. + pmbgpd: introduced a BGP Looking Glass server allowing to perform queries, ie. lookup of IP addresses/prefixes or get the list of BGP peers, against available BGP RIBs. The server is asyncronous and uses ZeroMQ as transport layer to serve incoming queries. Sample C/Python LG clients are available in 'examples/lg'. A sample LG server config is available in QUICKSTART. Request/Reply Looking Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'. + pmacctd: a single daemon can now listen for traffic on multiple interfaces via a polling mechanism. This can be configured via a pcap_interfaces_map feature (interface/pcap_interface can still be used for backward compatiblity to listen on a single interface). The map allows to define also ifindex mapping and capturing direction on a per-interface basis. The map can be reloaded at runtime via a USR2 signal and a sample map is in examples/pcap_interfaces.map.example. + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and kafka_partition_key knobs is introduced. The Kafka topic can contain variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which are all computed when data is purged to the backend. This feature is in addition to the existing kafka_partition feature which allows to rely on the built-in Kafka partitioning to assign data statically to one partition or rely dynamically on the default partitioner. The feature is courtesy by Corentin Neau / Codethink ( @weyfonk ). + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone represented as yyyy-MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives the timestamps_rfc3339 knob can be used to enable this feature (left disabled by default for backward compatibility). + timestamps_utc: new knob to decode timestamps to UTC timezone even if the Operating System is set to a different timezone. On the goods of running a system set to UTC please read Q18 of FAQS. + sfacctd: implemented mpls_label_top, mpls_label_bottom and mpls_stack_depth primitives decoded from sFlow flow sample headers. Thanks to David Barroso ( @dbarrosop ) for his support. + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131 (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX option packets (these IEs were already supported when passed in flow data). Also added support for IE 351 (dataLinkFrameSection) which carries the initial portion of a sampled raw packet headers (a-la sFlow). This was tested working against a Cisco NCS 5k platform. + nfprobe plugin: added a new nfprobe_dont_cache knob allowing to disable caching and summarisation of flows (essentially letting the NetFlow/IPFIX probe behave like a sFlow probe). + nfprobe plugin: added support for MPLS_LABEL_1, NetFlow v9/IPFIX IE 70; improved support for BGP next-hop IE 18 and 63. Also support for IE 130/131 vi NetFlow v9/IPFIX Options was added. + sfprobe plugin: added sfprobe_source_ip knob to define the local IP address from which sFlow datagrams are exported; improved support for BGP next-hop. + nfacctd, sfacctd, BGP, BMP, Streaming Telemetry daemons: on Linux, if supported, use SO_REUSEPORT for the listening socket (added to existing SO_REUSEADDR option). + nfacctd, sfacctd: introduced new 'export_proto_sysid' primitive to give visibility to NetFlow v5/v8 engine_id / NetFlow v9 source ID / IPFIX Obs Domain ID / sFlow agentSubID. + nfacctd, sfacctd: extended nDPI support to NetFlow v9/IPFIX packets with IE 315 (dataLinkFrameSection) and sFlow v5 packets with header section. + nfacctd, sfacctd: extended custom primitives definition framework, aggregate_primitives, to NetFlow v9/IPFIX packets with IE 315 (dataLinkFrameSection) and sFlow v5 sampled headers section. + nfacctd, sfacctd: added per-collector packets and bytes counts to stats emitted via SIGUSR1. Also the output was made more formal (so to be more easily parsed) and is documented in the UPGRADE notes. + nfacctd, pmacctd, sfacctd: pcap_savefile_delay feature introduced to sleep for the supplied amount of seconds before playing a given pcap_savefile. Useful, for example, to let BGP/BMP sessions come up so that routing data is available for correlation when processing data in the trace. + Kafka plugin: configuring statistics.interval.ms to a positive value in a kafka_config_file makes now librdkafka log plenty of internal metrics. + BGP daemon: added support for Extended BGP Administrative Shutdown Communication (draft-snijders-idr-rfc8203bis-00). + BMP daemon: added support for draft-ietf-grow-bmp-adj-rib-out-01 and draft-ietf-grow-bmp-loc-rib-01. As a result of that, Route Monitor log messages now contain indication of is_out and is_filtered. + BMP daemon: added support for stats reports 9, 10, 11, 12 and 13 and descriptions for the different Peer Types and and Peer Down reasons. Finally, indication of is_post is now making to Route Monitor log messages. + plugin_pipe_zmq: introduced plugin_pipe_zmq_hwm (high water mark) knob to control the maximum amount of messages than can be stored in the ZeroMQ queue. + [ns]facctd_allow_file: the map is now made reloadable at runtime via SIGUSR2 and accepts IPv4/IPv6 prefixes increasing its scale (before it was only accepting individual IP addresses). + pmacctd: added support for IPv6, MPLS for DLT_LINUX_SLL captures. Thanks to David Barroso ( @dbarrosop ) for his support. + uacctd: added a global 'direction' knob to give visibility of data capturing direction, ie. in/out. Useful for pre_tag_map use. + MySQL plugin: added sql_port knob in order to specify non-default ports for connecting to the database. Patch is courtesy by Vadim Tkachenko ( @vadimtk ). ! fix, plugins: getppid() parent process health check improved so to work in Docker environments not assuming anymore parent PID is 1. Patch is courtesy by Hidde van der Heide ( @hvanderheide ). ! fix, plugins: imposing a budget for received messages (100) so to preserve fairness of other operations (ie. time keeping, bucketing, reloading maps, etc.) and prevent starvations. ! fix, plugins: retry when zmq_getsockopt() for ZMQ_EVENTS returns EINTR. Thanks to Wouter de Jong for his support solving the issue. ! fix, plugins: when executing triggers, the first argument passed to execv() should be the path to the invoked executable to prevent execv(3) to fail and return EFAULT on OpenBSD. Patch is courtesy by @higgsd. ! fix, BGP daemon: improved support of multiple capabilities per optional parameter in the OPEN message. Also add-path capability is now advertised if neighbor supports send/receive (previously it was sent back on send only) of such capability. Thanks to Radu Anghel ( @cozonac ) for his support. ! fix, BGP daemon: upon route lookup, don't perform ADD-PATH logics if no PATH-ID (even if ADD-PATH capability is announced by the peer). Thanks to Camilo Cardona ( @jccardonar ) for his support solving the issue. ! fix, BGP daemon: wrong type 2 32-bit ASN Route Distinguisher was defined in network.h. Thanks to Thomas Graf for reporting the issue. ! fix, BGP, BMP daemons: lookup of BGP-LU entries is now performed against the correct RIB. ! fix, BMP daemon: the BMP thread is now made mutually exclusive with the BGP one (until an use-case needs to run them both). This is to potentially prevent BGP and BMP information to interfere with each other when correlated. Also the 'bmp' keyword was added for *_as and *_net config directives (ie. nfacctd_as, nfacctd_net). Thanks to Juan Camilo Cardona ( @jccardonar ) for his support. ! fix, BMP daemon: improved correlation of BMP data with traffic data by supporting a replication use-case (the BMP exporter is a route -server rather than an actual Edge Router) upon lookup. Thanks to Juan Camilo Cardona ( @jccardonar ) for his support. ! fix, BMP daemon: in bgp_peer_cmp() and bgp_peer_host_addr_cmp() the comparison function has been changed from generic memcmp() to a more specific host_addr_cmp() as paddings were giving issues. Thanks to Juan Camilo Cardona ( @jccardonar ) for reporting the issue. ! fix, BMP daemon: a pm_tdestroy call in bmp_peer_close() was leading to SEGV under certain conditions by not NULL'ing all pointers. Thanks to Juan Camilo Cardona ( @jccardonar ) for reporting the issue. ! fix, nfacctd: prevent time calculations to underflow in cases in which sysUptime < first or last flow switched timestamps in NetFlow v5. Patch is courtesy by David Steinn Geirsson ( @dsgwork ). ! fix, nfacctd: in the context of aggregate_primitives, now enforcing terminating the zero when decoding variable-length IEs when applying string semantics. ! fix, nfprobe: changed ifIndex fields from u_int16_t to u_int32_t in order to prevent overflows and aligning to the rest of structs. ! fix, MySQL plugin: minor code revisions to restore compiling against MariaDB 10.2. ! fix, sql_common.c: increased read_SQLquery_from_file() buffer size so that sql_table_schema can be fed with longer CREATE TABLE statements. ! fix, print, SQL plugins: post_tag, post_tag2 support was added to sql_table and print_output_file. Also for Kafka, RabbitMQ plugins kafka_topic and amqp_routing_key variables support was harmonized with print and SQL plugins (ie. $pre_tag renamed to $tag), see UPGRADE notes. ! fix, SQL plugins: sql_startup_delay was not being honored when sql_trigger_exec was defined without a sql_trigger_time resulting in empty environment variables being passed to the triggered script. Thanks to Johannes Maybaum for his support resolving the issue. ! fix, pkt_handlers.c: tmp_asa_bi_flow value was ignored when applied to a specific plugin. ! fix, util.c: when data timestamp is not available, dynamic file and table names variables were populated with a 1-Jan-1970 date. Now the current timestamp is used instead as last resort. Patch is courtesy by Ivan F. Martinez ( @ivanfmartinez ). ! fix, addr.c: host_addr_mask_sa_cmp() and str_to_addr_mask() network mask computation for IPv6 addresses was wrong. allow_file feature was affected. ! fix, build system: several patches committed to the build system to simplify libraries probing, make sure to bail out upon error. Also now a minimum required version is imposed to almost all libraries. - --enable-threads / --disable-threads: removed the configure switch that was allowing to compile pmacct even when no pthreads library was available on a system. From now on support for threads is mandatory. - BGP daemon: offline code, ie. bgp_daemon_offline_* config directives, has been deprecated in favor of other approaches, ie. BGP Looking Glass and BGP Xconnects. - pkt_len_distrib: the primitive, which was meant to bucket packet / flow / sample lengths in a distribution has been obsoleted. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
