VERSION. 1.7.4
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI data; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.4.tar.gz CHANGELOG. + Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder for multi-vendor environments written in Python3. It supports gRPC transport along with Protobuf encoding as input and can output to Kafka with Avro encoding. Output to files and JSON encoding is currently supported sending data via ZMQ to pmtelemetryd first. It was tested working with data input from Cisco and Huawei routers and v3 replaces v2. Thanks to the Streaming Telemetry core team: Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ), Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ). + Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.) have been wired to the VXLAN decoding and new ones (tunnel_src_mac, tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined. + BMP daemon: added support for Peer Up message namespace for TLVs (draft-ietf-grow-bmp-peer-up) and also support for Route Monitor and Peer Down TLVs (draft-ietf-grow-bmp-tlv). + BGP, BMP daemons: in addition to existing JSON export, data can now be exported in Apache Avro format. There is also support for the Confluent Schema Registry. + Introduced support for JSON-encoded Apache Avro encoding. While the binary-encoded Apache Avro is always recommended for any production scenarios (also to optionallly leverage Confluent Schema Registry support), JSON-encoded is powerful for testing and troubleshooting scenarios. + sfprobe plugin: added support for IPv6 transport for sFlow export. sfprobe_agentip is an IP address put in the header of the sFlow packet. If underlying transport is IPv6, this must be configured to an IPv6 address. + zmq_common.[ch]: Improved modularity of the ZMQ internal API and decoupled bind/connect from push/pull and pub/sub; also improved support for inproc sockets. All to increase the amount of use-cases covered by the API. + bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use cases. + nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to override ifindexes dynamically determined (ie. by NFLOG) with values computed by [sn]fprobe_ifindex. + MySQL, PostgreSQL plugins: added support for SSL/TLS connections by specifying a CA certificate (sql_conn_ca_file). + Kafka, AMQP plugins: amqp_markers and kafka_markers have now been properly re-implemented when output encoding is Avro using an own Avro schema (instead of squatting pieces of JSON in the data stream for the very purpose). + print plugin: introduced print_write_empty_file config knob (true, false) to create an empty output file when there are no cache entries to purge. Such behaviour was present in versions up to 0.14 and may be preferred by some to the new >= 1.5 versions behaviour. Thanks to Lee Yongjae ( @setup74 ) for the contribution. ! fix, signals.c: signals handling has been restructured in order to block certain signals during critical sections of data processing. Thanks to Vaibhav Phatarpekar ( @vphatarp ) for the contribution. ! fix, signals.c: slimmed reload() signal handler code and moved it to a synchronous section. The handler is to reset logging output to files or syslog. Thanks to Jared Mauch ( @jaredmauch ) for his support resolving this. ! fix, pmbgpd, pmbmpd and pmtelemetryd daemons: added extra signals handling (SIGINT, SIGTERM, SIGCHLD) consistently to traffic daemons. ! fix, BGP daemon: withdrawals of label-unicast (support introduced in 1.7.3) and mpls-vpn NLRIs did fail to parse in release 1.7.3 and were silently discarded. ! fix, nfacctd: wired (BGP, BMP, ISIS, etc) lookups to NetFlow (Secure) Event Logging (NEL/NSEL). ! fix, pmtelemetryd: re-implemented a decoder for so-called Cisco v1 Streaming Telemetry proprietary header over UDP/TCP streams. ! fix, pmtelemetryd: improved sanitization of input JSON objects by also checking for isspace() other than isprint() for pretty-printed objects. ! maps_index: optimized lookups, improved debugging output upon loading indexes. ! fix, tee plugin: overwriting computed IP address length with socket container length was found to prevent output data on some BSDs. ! fix, kafka_common.c: if taking the p_kafka_close() route, ensure to return and not perform any further polling in order to avoid SEGVs. ! fix, BMP daemon: incorrect decoding of type was preventing correct logging of Init and Term messages extra info. Also in Term messages TLV data was incorrectly consumed twice triggering length check warnings. ! fix, BMP daemon: added checks for successful BGP PDU parsing in both Peer Up (BGP OPEN) and Route Monitor (BGP UPDATE) messages. ! fix, BMP daemon: improved length checks and making sure that strings potentially non null-terminated are now terminated. Also TLV-related code has been refactored. ! fix, pmbgp.py: the example client for BGP Looking Glass was migrated to Python3: thanks to @brusilov for the contribution. ! fix, nfacctd: if src_port or dst_port primitives are selected, enable IP fragment handling. Needed to process L4 of IPFIX IE #351. ! fix, nfv9_template.c: correct handling of variable-length IPFIX fields. Thanks to Nimrod Mesika ( @nimrody ) for the contribution. ! fix, PostgreSQL plugin: ABSTIME was replaced with to_timestamp() in queries as support for ABSTIME was dropped as of PostgreSQL 12. Many thanks to Manuel Mendez ( @mmlb ) for the contribution. ! fix, PostgreSQL plugin: SEGVs were observed when the queue of pending queries was non-empty (ie. nfacctd_time_new set to false, default); thanks to Guo-Wei Su ( @nansenat16 ) for the contribution. ! fix, cfg_handlers: [sn]facctd_disable_checks, nfacctd_disable_opt_ scope_check could not be properly set to false. ! fix, sql_common.c: src_host_coords and dst_host_coords primitives have been correctly spaced in SQL queries. Also float values are now quoted. Finally, sampling_direction primitive is encoded correctly. ! fix, kafka plugin: if kafka_avro_schema_registry is in use, subject name is aligned to Kafka topic name (if topic is not dynamic). ! fix, pretag.c: when using 'label', store the label string in the heap (instead of the stack). Thanks to Raphael P. Barazzutti ( @rbarazzutti ) for the contribution. ! fix, pretag.c: JEQ labels are now correctly free() during init upon map reload. ! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was causing plugin_pipe_zmq operations to fail on certain compilers (ie. gcc7). Thanks to Yuri Lachin ( @yuyutime ) for his support. ! fix, cfg_handlers.c: reviewed handling of parsed 'zero' value for several config directives. ! fix, countless code warnings when enabling -Wall (--enable-debug); also included -Wall in Continuous Integration tests. Restructured globals, header inclusions, function prototypes definition, etc. Many thanks to Marc Sune ( @msune ) for all his efforts. ! fix, configure.ac: evaluation of --enable-debug pushed to the end of the script so to not interfere with tests (ie. alignment, endianess, etc.). - BMP daemon: retired support for draft-hsmit-bmp-extensible-routemon- -msgs. - AMQP plugin: obsoleted amqp_avro_schema feature (which includes amqp_avro_schema_routing_key and amqp_avro_schema_refresh_time keys config keys). Avro schemas can now only be written to files. 1.7.3 -- 16-05-2019 + Introduced the RPKI daemon to build a ROA database and check prefixes validation status and coverages. Resource Public Key Infrastructure (RPKI) is a specialized public key infrastructure (PKI) framework designed to secure the Internet routing. RPKI uses certificates to allow Local Internet Registries (LIRs) to list the Internet number resources they hold. These attestations are called Route Origination Authorizations (ROAs). ROA information can be acquired in one of the two following ways: 1) importing it using the rpki_roas_file config directive from a file in the RIPE Validator format or 2) connecting to a RPKI RTR Cache for live ROA updates; the cache IP address/port being defined by the rpki_rtr_cache config directive (and a few more optional rpki_rtr_* directives are available and can be reviwed in the CONFIG-KEYS doc). The ROA fields will be populated with one of these five values: 'u' Unknown, 'v' Valid, 'i' Invalid no overlaps, 'V' Invalid with a covering Valid prefix, 'U' Invalid with a covering Unknown prefix. Thanks to Job Snijders ( @job ) for his support and vision. + Introducing pmgrpcd.py, written in Python, a daemon to handle gRPC- based Streaming Telemetry sessions and unmarshall GPB data. Code was mostly courtesy by Matthias Arnold ( @tbearma1 ). This is in addition (or feeding into) pmtelemetryd, written in C, a daemon to handle TCP/UDP-based Streaming Telemetry sessions with JSON-encoded data. Thanks to Matthias Arnold ( @tbearma1 ) and Thomas Graf for their support and contributing code. + pmacctd, uacctd: added support for CFP (Cisco FabricPath) and Cisco Virtual Network Tag protocols. Both patches were courtesy by Stephen Clark ( @sclark46 ). + print plugin: added 'custom' to print_output. This is to cover two main use-cases: 1) use JSON or Avro encodings but fix the format of the messages in a custom way and 2) use a different encoding than JSON or Avro. See also example in examples/custom and new directives print_output_custom_lib and print_output_custom_cfg_file. The patch was courtesy by Edge Intelligence ( @edge-intelligence ). + Introducing mpls_pw_id aggregation primitive and mpls_pw_id key in pre_tag_map to filter on signalled L2 MPLS VPN Pseudowire IDs. + BGP daemon: added bgp_disable_router_id knob to enable/disable BGP Router-ID check, both at BGP OPEN time and BGP lookup. Useful, for example, in scenarios with split BGP v4/v6 AFs over v4/v6 transports. + BGP, BMP daemons: translate origin attribute numeric value into IGP (i), EGP (e) and Incomplete (u) strings. + plugins: added new plugin_exit_any feature to make the daemon bail out if any (not all, which is the default behaviour) of the plugins exits. + maps_index: improved selection of buckets for index hash structure by picking the closest prime number to the double of the entries of the map to be indexed in order to achieve better elements dispersion and hence better performances. + nfacctd: added support for IPFIX templateId-scoped (IE 145) sampling information. + pmacctd, uacctd, sfacctd, nfacctd: added a -M command-line option to set *_markers (ie. print_markers) to true and fixed -A command-line option to set print_output_file_append to align to true/false. ! fix, BGP, BMP, Streaming Telemetry daemons: improved sequencing of dump events by assigning a single sequence number per event (ie. for streaming pipeline scenarios in order to reduce correlation with dump_init/dump_close messages). Also amount of record dumped was added to the close message. ! fix, BGP, BMP, Streaming Telemetry daemons: removed hierarchical json_decref() since json_object_get() borrows reference. This was occasionaly leading to SEGVs. ! fix, uacctd: dynamically allocate jumbo_container buffer size as packets larger than 10KB, previous static allocation, would lead to crashes. ! fix, nfacctd: wired (BGP, BMP, ISIS, etc.) lookups to the NEL/NSEL use-case. ! fix, nfacctd: search for IE 408 (dataLinkFrameType) was leading to SEGVs. Also improved handling of variable-length IPFIX templates. ! fix, BMP daemon: solved an occasional truncation of the last message in a packet. ! fix, BGP daemon: when processing bgp_daemon_md5_file, ipv4 addresses were incorrectly translated to ipv4-mapped ipv6 ones as a result of which TCP-MD5 hashes were not correctly bound to sockets. ! fix, BGP daemon: improved label-unicast and mpls-vpn SAFIs handling (some bogus messages, multiple labels, etc.). ! fix, BGP daemon: introduced PREFIX_STRLEN to make enough room for prefix2str() calls (before unsufficient INET6_ADDRSTRLEN was used). ! fix, BMP daemon: improved handling of ADD-PATH capability. ! fix, plugins: an incorrect evaluation in P_cache_attach_new_node did make possible to buffer overrun in plugins cache allocation. This was found related to a "[..]: Assertion `!cache_ptr->stitch' failed." daemon bail-out message. ! fix, plugins: if pidfile directive was enabled, exit_gracefully() was mistakenly deleting the plugin pidfile when called by a child process (ie. writer, dumper, etc.). ! fix, plugins: when taking exit_gracefully(), if the process is marked as 'is_forked', just exit and don't perform extra ops in exit_all() or exit_plugin(). ! fix, plugins: re-evaluate dynamic tables/files name if *_refresh_time is different than *_history period. ! fix, SQL plugins: a missing 'AND' was making SQL statements related to src_host_coords and dst_host_coords fail. ! fix, GeoIPv2: if no match is returned by libmaxminddb, return O1 code (Other Country) instead of a null value. ! fix, flow_to_rd_map: mpls_vpn_id was not working when maps_index was enabled. Also partly re-written mpls_vpn_id handler. ! fix, nfprobe plugin: serialize_bin() function introduced for correct serialization of custom primitives defined with 'raw' semantics. ! fix, PostgreSQL plugin: testing for presence of PQlibVersion() in libpq to prevent compiling issues (ie. on CentOS 6). ! fix, MySQL plugin: including mysql_version.h to compile successfully against newer MariaDB releases. ! fix, nDPI classification: send log message if 'class' primitive is selected but nDPI is not compiled in; also updated code to follow API changes in versions >= 2.6 of the library. Dropped support for versions < 2.4. ! fix, sfprobe plugin: added (and documented) conditional for optional export of classification info. ! fix, aggregate_primitives: field_type is now also allowed for pmacctd and uaccd daemons so that it can be used for NetFlow v9/IPFIX export (nfprobe plugin) purposes. ! fix, pre_tag_map: if no 'ip' keyword is specified, an entry of the map gets recirculated in order to be set for both v4 and v6 maps. If a 'set_label' is also specified, it was causing a SEGV. Now the label is correctly copied in case of recirculation. ! fix, zmq_common.c: added option for non-blocking p_zmq_send_bin() as otherwise program would block in case of no consumers (main use-case: flow replication over ZeroMQ queues); as a result, a generous hwm value was added on both sides of these queues. ! fix, zmq_common.c: ZAP socket moved inside thread to prevent failed assert() when compiling with gcc7/gcc8. Also a single user/password auto-generated combination is used for all plugins. ! fix, signals.c: SIGUSR1 handler for nfacctd and nfacctd is changed to syncronous in order to prevent race conditions. Also, in pmacctd, upon sending SIGUSR1, stats were not printed when reading packets from a pcap_interfaaces_map. ! fix, plugin_cmn_json.c: if leaving protocols numerical (ie. proto, tunnel_proto primitives), convert them to string-represented numbers for data consistency for consumers. ! fix, util.c: open_output_file(), if file exists and it's a FIFO then set O_NONBLOCK when opening. ! fix, pretag.c: pretag_index_report() was reporting incorrect info of the hash structure built for the maps_index feature. Its format was has also changed to be better parseable. ! fix, compile time warnings: several warnings were addressed including but not restricted to -Wformat ones. Also an annotation was added to the Log function to inform the compiler it's a printf-style function, allowing it to give warnings for argument mismatches. - --enable-ipv6 configure script switch has been deprecated and, as a result, IPv6 support was made mandatory. - BGP daemon: removed unused pathlimit field from bgp_attr structure. - pmacct client: removed deprecated SYM field from from formatted and CSV headers. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists