VERSION. 1.6.0
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via streamed telemetry. A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP BMP and streamed telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.6.0.tar.gz CHANGELOG. + Streamed telemetry daemon: quoting Cisco IOS-XR Telemetry Configuration Guide at the time of this writing: "Streaming telemetry [ .. ] data can be used for analysis and troubleshooting purposes to maintain the health of the network. This is achieved by leveraging the capabilities of machine-to-machine communication. [ .. ]" Streamed telemetry support comes in two flavours: 1) a telemetry thread can be started in existing daemons, ie. sFlow, NetFlow/IPFIX, etc. for the purpose of data correlation and 2) a new daemon pmtelemetryd for standalone consumpton of data. Streamed telemetry data can be logged real-time and/or dumped at regular time intervals to flat-files, RabbitMQ or Kafka brokers. + BMP daemon: introduced support for Route Monitoring messages. RM messages "provide an initial dump of all routes received from a peer as well as an ongoing mechanism that sends the incremental routes advertised and withdrawn by a peer to the monitoring station". Like for BMP events, RM messages can be logged real-time and/or dumped at regular time intervals to flat-files, RabbiMQ and Kafka brokers. RM messages are also saved in a RIB structure for IP prefix lookup. + uacctd: ULOG support switched to NFLOG, the newer and L3 independent Linux packet logging framework. One of the key advantages of NFLOG is support for IPv4 and IPv6 (whereas ULOG was restricted to IPv4 only). The code has been contributed by Vincent Bernat ( @vincentbernat ). + build system: it was modernized so not to rely on specific and old versions of automake and autoconf, as it was the case until 1.5. Among the things, pkg-config and libtool are leveraged and an autogen.sh script is generated. The code has been contributed by Vincent Bernat ( @vincentbernat ). + sfacctd: RabbitMQ and Kafka support was introduced to real-time log and/ or dump at regular time intervals of sFlow counters. This is in addition to existing support for flat-files. + maps_index: several improvements were carried out in the area of indexing of maps: optimizations to pretag_index_fill() and pretag_index_lookup() to improve lookup speeds; optimized id_entry structure, ie. by splitting key and non-key parts, and hashing key in order to consume less memory; added duplicate entry detection (cause of sudden index destruction); pretag_index_destroy() destroys hash keys for each index entry, solving a memory leak issue. Thanks to Job Snijders ( @job ) for his support. + Introduced 'export_proto_seqno' aggregation primitive to report on sequence number of the export protocol (ie. NetFlow, sFlow, IPFIX). This feature may enable more advanced offline analysis of packet loss, out of orders, etc. over time windows than basic online analytics provided by the daemons. + log.c: logging moved from standard output (stdout) to standard error (stderr) so to not conflict with stdout printing of statistics (print plugin). Thanks to Jim Westfall ( @jwestfall69 ) for his support. + print plugin: introduced a new print_output_lock_file config directive to lock standard output (stdout) output so to prevent multiple processes (instances of the same print plugin or different instances of print plugin) overlap output. Thanks to Jim Westfall ( @jwestfall69 ) for his support. + pkt_handlers.c: euristics in NetFlow v9/IPFIX VLAN handler were improved for the case of flows in egress direction. Also IP protocol checks were removed for UDP/TCP ports and TCP flags in case the export protocol is NetFlow v9/IPFIX. Thanks to Alexander Ponamarchuk for his support. ! Code refactoring: improved re-usability of much of the BGP code (so to make it possible to use it as a library for some BMP daemon features, ie. Route Monitoring messages support); consolidated functions to handle log and print plugin output files; improved log messages to always include process name and type. ! fix, bpf_filter.c: issue compiling against libpcap 1.7.x; introduced a check for existing bpf_filter() in libpcap in order to prevent namespace conflicts. ! fix, tmp_net_own_field default value changed to true. This knob can be still switched to false for this release but is going to be removed soon. ! fix, cfg.c, cfg_handlers.c, pmacct.c: some configuration directives and pmacct CL parameters requiring string parsing, ie. -T -O -c, are now passed through tolower(). ! fix, MongoDB plugin: removed version check around mongo_create_index() and now defaulting to latest MongoDB C legacy driver API. This is due to some versioning issue in the driver. ! fix, timestamp_arrival: primitive was reporting incorrect results (ie. always zero) if timestamp_start or timestamp_end were not also specified as part of the same aggregation method. Many thanks to Vincent Morel for reporting the issue. ! fix, thread stack: a value of 0, default, leaves the stack size to the system default or pmacct minimum (8192000) if system default is too low. Some systems may throw an error if the defined size is not a multiple of the system page size. ! fix, nfacctd: improved NetFlow v9/IPFIX parsing. Added new length checks and fixed some existing checks. Thanks to Robert Wuttke ( @Benocs ) for his support. ! fix, pretag_handlers.c: BPAS_map_bgp_nexthop_handler() and BPAS_map_bgp_ peer_dst_as_handler() were not setting a func_type. ! fix, JSON support: Jansson 2.2 does not have json_object_update_missing() function which was introduced in 2.3. This is not provided as part of a jansson.c file and compiled in conditionally, if needed. Jansson 2.2 is still shipped along by some recent OS releases. Thanks to Vincent Bernat ( @vincentbernat ) for contributing the patch. ! fix, log.c: use a format string when calling syslog(). Passing directly a potentially uncontrolled string could crash the program if the string contains formatting parameters. Thanks to Vincent Bernat ( @vincentbernat ) for contributing the patch. ! fix, sfacctd.c: default value for config.sfacctd_counter_max_nodes was set after sf_cnt_link_misc_structs(). Thanks to Robin Douine for his support resolving the issue. ! fix, sfacctd.c: timestamp was consistently being reported as null in sFlow counters output. Thanks to Robin Douine for his support resolving the issue. ! fix, SQL plugins: $SQL_HISTORY_BASETIME environment variable was reporting a wrong value (next basetime) in the sql_trigger_exec script. Thanks to Rain Nõmm for reporting the issue. ! fix, pretag.c: in pretag_index_fill(), replaced memcpy() with hash_dup_key() also a missing res_fdata initialization in pretag_index_lookup() was solved; these issues were originating false negatives upon lookup. Thanks to Rain Nõmm fo his suppor. ! fix, ISIS daemon: hash_* functions renamed into isis_hash_* to avoid name space clashes with their BGP daemon counter-parts. ! fix, kafka_common.c: rd_kafka_conf_set_log_cb moved to p_kafka_init_host() due to crashes seen in p_kafka_connect_to_produce(). Thanks to Paul Mabey for his support resolving the issue. ! fix, bgp_lookup.c: bgp_node_match_* were not returning any match in bgp_follow_nexthop_lookup(). Thanks to Tim Jackson ( @jackson-tim ) for his support resolving the issue. ! fix, sql_common.c: crashes observed when nfacctd_stitching was set to true and nfacctd_time_new was set to false. Thanks to Jaroslav Jiráse ( @jjirasek ) for his support solving the issue. - SQL plugins: sql_recovery_logfile feature was removed from the code due to lack of support and interest. Along with it, also pmmyplay and pmpgplay tools have been removed. - pre_tag_map: removed support for mpls_pw_id due to lack of interest. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
