VERSION. 1.7.7
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI data; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Collected data can be easily exported (ie. via Kafka) to modern databases like ElasticSearch, Apache Druid and ClickHouse and (ie. via flat-files) to classic tools Cacti, RRDtool and MRTG, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.7.tar.gz CHANGELOG. + BGP, BMP, Streaming Telemetry daemons: introduced parallelization of dump events via a configurable amount of workers where the unit of parallelization is the exporter (BGP, BMP, telemetry exporter), ie. in a scenario where there are 4 workers and 4 exporters each worker is assigned one exporter data to dump. + pmtelemetryd: added support for draft-ietf-netconf-udp-notif: a UDP-based notification mechanism to collect data from networking devices. A shim header is proposed to facilitate the data streaming directly from the publishing process on network processor of line cards to receivers. The objective is a lightweight approach to enable higher frequency and less performance impact on publisher and receiver process compared to already established notification mechanisms. Many thanks to Alex Huang Feng ( @ahuangfeng ) and the whole Unyte team. + BGP, BMP, Streaming Telemetry daemons: now correctly honouring the supplied Kafka partition key for BGP, BMP and Telemetry msg logs and dump events. + BGP, BMP daemons: a new "rd_origin" field is added to output log/ dump to specify the source of Route Distinguisher information (ie. flow vs BGP vs BMP). + pre_tag_map: added ability to tag new NetFlow/IPFIX and sFlow sample_type types: "flow-ipv4", "flow-ipv6", "flow-mpls-ipv4" and "flow-mpls-ipv6". Also added a new "is_bi_flow" true/false key to tag (or exclude) NSEL bidirectional flows. Added as well a new "is_multicast" true/false config key to tag (or exclude) IPv4/IPv6 multicast destinations. + maps_index: enables indexing of maps to increase lookup speeds on large maps and/or sustained lookup rates. The feature has been remplemented using stream-lined structures from libcdada. This is a major work that helps preventing the unpredictable behaviours caused by the homegrown map indexing mechanism. Many thanks to Marc Sune ( @msune ). + maps_index: support for indexing src_net and dst_net keywords has been added. + Added <daemon_name>_ipv6_only config directives to optionally enable the IPV6_V6ONLY socket option. Also changed the wrong setsockopt() IPV6_BINDV6ONLY id to IPV6_V6ONLY. + Added log function to libserdes to debug transactions with the Schema Registry when kafka_avro_schema_registry is set. + nDPI: newer versions of the library (ie. >= 3.5) bring changes to the API. pmacct is now aligned to compile against these. + pmacctd: added pcap_arista_trailer_offset config directive since Arista has changed the structure of the trailer format in recent releases of EOS. Thanks to Jeremiah Millay ( @floatingstatic ) for his patch. + More improvements carried out on the Continuous Integration (CI) side by migrating from Travis CI to GitHub Actions. Huge thanks to Marc Sune ( @msune ) to make all of this possible. + More improvements also carried out in the space of the Docker images being created: optimized image size and a better layered pipeline. Thanks to Marc Sune ( @msune ) and Daniel Caballero ( @dcaba ) to make all of this possible. + libcdada shipped with pmacct was upgraded to version 0.3.5. Many thanks Marc Sune ( @msune ) for his work with libcdada. ! build system: several improvements carried out in this area, ie. improved MySQL checks, introduced pcap-config tool for libpcap, compiling on BSD/old compilers, etc. Monumental thanks to Marc Sune ( @msune ) for his continued help. ! fix, nfacctd: improved euristics to support the case of flows with both IPv4 and IPv6 source / destination addresses (either or populated). Also improved euristics to distinguish event data vs traffic data in NetFlow v9/IPFIX from Cisco 9300/9500, ASA firewalls and Cisco 4500X. ! fix, nfacctd: improved support for initiatorOctets (IE #231) and responderOctets (IE #232). Thanks to Esben Laursen ( @hyberdk ) for reporting the issue. ! fix, nfacctd: in NF_mpls_vpn_id_handler() double ntohl() calls were applied for the case of 'vrfid'-encoded mpls_vpn_rd field. ! fix, sfacctd: wrong ethertype set for VLAN-tagged, MPLS-labelled IPv6 traffic. Impacting BGP resolution among others. Thanks to Jeremiah Millay ( @floatingstatic ) for his help resolving the problem. ! fix, BGP, BMP daemons: parsing improvements: added a check for BGP Open message and BGP Open Options lengths. Strengthened parsing of Peer Up, Route Monitoring and Peer Down v4 messages. ! fix, BGP, BMP daemon: when using Avro encoding and Avro Schema Registry, attempt to reconnect if serdes schemas are voided. Also now checking for serdes schema definitions before doing a serdes_schema_serialize_avro() to avoid triggering a SEGV. Finally improved serdes logging. ! fix, BGP, Streaming Telemetry daemons: in daemon logs, summary counters for amount of tables / entries dumped were wrong. ! fix, BGP daemon: distinguish among null and zero value AIGP and Prefix SID attributes. Same applies for Local Preference and MED attributes. ! fix, BMP daemon: resolved a memory leak in bgp_peers_free(). Thanks to Pether Pothier ( @pothier-peter ) for his patch. Also resolved a leak caused by an invalid BGP message contained in a BMP Route Message v4. ! fix, BMP daemon: correctly setting peer_ip and peer_tcp_port JSON fields for Term messages. Also the correct bmp_router value when bmp_daemon_parse_proxy_header feature is enabled. ! fix, BMP daemon: several encoding issues when using Apache Avro ie. u_int64_t now correctly encoded with avro_value_set_long(), certain u_int32_t fields switched to avro_value_set_long() due to lack of unsignedness in Avro encoding, improved various aspectes of Avro-JSON format output, etc. ! fix, pmtelemetryd: wrong parsing of pm_tfind() output was leading to mistaken data attribution of UDP-based peers (always first peer to connect was being picked). ! fix, pmtelemetryd: when set, the pidfile config directive was not being correctly honoured. ! fix, RPKI: the RTR PDU element for maxLength is uint8, therefore it might have been possible to transmit incorrect RTR data. Thanks to Job Snijders ( @job ) for his patch. ! fix, SQL plugins: amended the text composition of SQL queries that are involving latitude and longitude keys. ! fix, MySQL plugin: check for 'unix:' prefix string only when a sql_host configuration directive is specified. ! fix, nfprobe: modernized Application Information export. Until the previous release pmacct was adhering to aging NBAR model whereas now NBAR2 has been implemented. Thanks to Rob Cowart ( @robcowart ) for helping out resolving this issue. ! fix, tee plugin: restored usefulness of tee_source_ip which was broken in 1.7.6. Thanks to Jeremiah Millay ( @floatingstatic ) for reporting the issue. ! fix, maps_index: indexing of mpls_pw_id was broken. Also now, when the feature is enabled, actual data is being referenced in the index structure instead of creating a copy of it; thanks to Sander van Delden ( @SanderDelden ) for reporting the memory leak that was resulting from the copy. ! fix, kafka_common.c: solved memory leak in p_kafka_set_topic() when Kafka session was getting in down state. Many thanks to Peter Pothier ( @pothier-peter ) for nailing the issue. ! fix, net_aggr.[ch]: when a networks_file is specified in the config, gracefully handle max memory structure depth; added also de-duplication of entries. ! fix, pmacct-defines.h: if PCAP_NETMASK_UNKNOWN is not defined, ie. in libpcap < 1.1.0, let's define it. ! fix, SO_REUSEPORT feature was being restricted to Linux only in previous releases: now it has been unlocked to all other OS that do support the feature. ! fix, split SO_REUSEPORT and SO_REUSEADDR setsockopt() calls. Thanks to @eduarrrd for reporting and resolving the issue. ! fix, several code warnings catched gcc9 and clang. - Obsoleted sql_history_since_epoch, pre_tag_map_entries and refresh_maps configuration directives. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
