VERSION. 1.6.1
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.6.1.tar.gz CHANGELOG. + Introduced pmbgpd daemon: a stand-alone BGP collector daemon; acts as a passive neighbor and maintains per-peer RIBs; can log real-time and/or dump at regular time-intervals BGP data to configured backends. + Introduced pmbmpd daemon: a stand-alone BMP collector daemon; can log real-time and/or dump at regular time-intervals BMP and BGP data to configured backends. + Introduced Apache Avro as part of print, AMQP and Kafka output: Apache Avro is a data serialization system providing rich data structures, a compact, fast, binary data format, a container file to store persistent data, remote procedure call (RPC) and simple integration with dynamic languages. The implementation is courtesy by Codethink Ltd. + as_path, std_comm and ext_comm primitives: along with their src counter parts, ie. src_as_path etc., have been re-worked to a variagle-length internal representation which will lead, when using BGP primitives, to memory savings of up to 50% compared to previous releases. + std_comm, ext_comm primitives: primitives are de-coupled so that they are not multiplexed anymore in the same field, on output. Added a tmp_comms_same_field config directive for backward compatibility. + nfacctd: added support for repeated NetFlow v9/IPFIX field types. Also flowStartDeltaMicroseconds (IE #158) and flowEndDeltaMicroseconds (#159) are now supported for timestamping. + kafka plugin: it is now possible to specify -1 (RD_KAFKA_RTITION_UA) as part of the kafka_partition config directive. Also, introduced support for Kafka partition keys via kafka_partition_key and equivalent config directives. + kafka plugin: kafka_broker_host directive now allows to specify multiple brokers, ie. "broker1:10000,broker2". The feature relies on capabilities of underlying rd_kafka_brokers_add(). + tee, nfprobe, sfprobe plugins: introduced Kafka support for internal pipe and buffering, ie. plugin_pipe_kafka. This is in addition to the existing support for homegrown internal buffering and RabbitMQ. + tee plugin: introduced support for variable-length buffers which reduces CPU utilization. + print, MongoDB, AMQP and Kafka plugins: re-worked max_writers feature to not rely anymore on waitpid() inside signal handlers as it was failing on some OS versions (and could not be reproduced on others). Thanks to Janet Sullivan for her support. + bgp_follow_nexthop_external: introduced feature to return, when true, the next-hop from the routing table of the last node part of the supplied IP prefix(es) as value for the 'peer_ip_dst' primitive. When false, default, it returns the IP address of the last node part of the bgp_follow_nexthop config key. + pmtelemetryd: added initial support for GPB. Input GPB data is currently base64'd in the telemetry_data field of the daemon output JSON object. + pmtelemetryd: Added telemetry statistics. For each peer, track the number of packets received, how many bytes are pulled off the wire, and the resulting message payload. Dump these counts in logdump. Patch is courtesy by Tim LaBerge. + amqp_markers, kafka_markers: added start/end markers feature to AMQP and Kafka plugins output same as for the print plugin (print_markers). + pre_tag_map: 'direction' keyword now applies to sFlow too: it does expect values 0 (ingress direction) or 1 (egress direction), just like before. In sFlow v2/v4/v5 this returns a positive match if: 1) source_id equals to input interface and this 'direction' key is set to '0' or 2) source_id equals to output interface and this 'direction' key is set to '1'. + bgp_agent_map: introduced support for input and output interfaces. This is relevant to VPN scenarios. + tmp_asa_bi_flow hack: bi-flows use two counters to report counters, ie. bytes and packets, in forward and reverse directions. This hack (ab)uses the packets field in order to store the extra bytes counter. ! fix, nfacctd: debugging NetFlow v9/IPFIX templates, added original field type number to the output when the field is known and its description is presented. ! fix, Jansson: added JSON_PRESERVE_ORDER flag to json_dumps() to give output consistency across runs. ! fix, kafka_common.c: added rd_kafka_message_destroy() to p_kafka_consume_ _data() to prevent memory leaks. Thanks to Paul Mabey for his support solving the issue. ! fix, kafka_common.c: p_kafka_set_topic() now gives it some time for the topic to get (auto) created, if needed. ! fix, print plugin: improved check for when to print table title (csv, formatted). Either 1) print_output_file_append is set to false or 2) print_output_file_append is set to true and file is to be created. ! fix, print_markers: start marker is now printed also in the case where print_output_file_append is set to true. Also, markers are now printed as a JSON object, if output is set to JSON. ! fix, pkt_handlers.c: removed l3_proto checks from NF_peer_dst_ip_handler() for cases where a v6 flows has a v4 BGP next-hop (ie. vpnv6) ! fix, pre_tag_map: removed 32 chars length limit from set_label statement. ! fix, custom primitives: names are now interpreted as case-insensitive. Patch is courtesy by Corentin Neau. ! fix, BGP, BMP and Streaming Telemetry: if reopening [bgp, bmp, telemetry]_ daemon_msglog_file via SIGHUP, reset reload flag. ! fix, BGP, BMP and Streaming Telemetry: removed gettimeofday() from bgp_ peer_dump_init() and bgp_peer_dump_close() in order to maintain a single timestamp for a full dump event. Thanks to Tim LaBerge for his support. ! fix, BGP, BMP and Streaming Telemetry: output log and dump messages went through a general review to improve information consistency and usability. Message formats are now documented in docs/MSGLOG_DUMP_FORMATS so to more easily track future changes. ! fix, pmtelemetryd: avoiding un-necessary spawn of a default plugin if none is defined. ! fix, pmtelemetryd: Mask SIGCHLD during socket IO. If we happen to be blocked in recv() while a log dump happens, recv() will fail with EINTR. This is to mask SIGCHLD during socket IO and restores the original mask after the IO completes. Patch is courtesy by Tim LaBerge. ! fix, build system: misc improvements made to the build system introduced in 1.6.0. Thanks to Vincent Bernat for his support in this area. ! fix, compiler warnings: ongoing effort to suppress warning messages when compiling. Thanks to Tim LaBerge, Matin Mitchell for their contributions. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
