VERSION. 1.7.0
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.0.tar.gz CHANGELOG. + ZeroMQ integration: by defining plugin_pipe_zmq to 'true', ZeroMQ is used for queueing between the Core Process and plugins. This is in alternative to the home-grown circular queue implementation (ie. plugin_pipe_size). plugin_pipe_zmq_profile can be set to one value of { micro, small, medium, large, xlarge } and allows to select among a few standard buffering profiles without having to fiddle with plugin_buffer_size. How to compile, install and operate ZeroMQ is documented in the "Internal buffering and queueing" section of the QUICKSTART document. + nDPI integration: enables packet classification, replacing existing L7-layer project integration, and is available for pmacctd and uacctd. The feature, once nDPI is compiled in, is simply enabled by specifying 'class' as part of the aggregation method. How to compile install and operate nDPI is documented in the "Quickstart guide to packet classification" section of the QUICKSTART document. + nfacctd: introduced nfacctd_templates_file so that NetFlow v9/IPFIX templates can be cached to disk to limit the amount of lost packets due to unknown templates when nfacctd (re)starts. The implementation is courtesy by Codethink Ltd. + nfacctd: introduced support for PEN on IPFIX option templates. This is in addition to already supported PEN for data templates. Thanks to Gilad Zamoshinski ( @zamog ) for his support. + sfacctd: introduced new aggregation primitives (tunnel_src_host, tunnel_dst_host, tunnel_proto, tunnel_tos) to support inner L3 layers. Thanks to Kaname Nishizuka ( @__kaname__ ) for his support. + nfacctd, sfacctd: pcap_savefile and pcap_savefile_wait were ported from pmacctd. They allow to process NetFlow/IPFIX and sFlow data from previously captured packets; these also ease some debugging by not having to resort anymore to tcpreplay for most cases. + pmacctd, sfacctd: nfacctd_time_new feature has been ported so, when historical accounting is enabled, to allow to choose among capture time and time of receipt at the collector for time-binning. + nfacctd: added support for NetFlow v9/IPFIX field types #130/#131, respectively the IPv4/IPv6 address of the element exporter. + nfacctd: introduced nfacctd_disable_opt_scope_check: mainly a work around to implementations not encoding NetFlow v9/IPIFX option scope correctly, this knob allows to disable option scope checking. Thanks to Gilad Zamoshinski ( @zamog ) for his support. + pre_tag_map: added 'source_id' key for tagging on NetFlow v9/IPFIX source_id field. Added also 'fwdstatus' for tagging on NetFlow v9/ IPFIX information element #89: this implementation is courtesy by Emil Palm ( @mrevilme ). + tee plugin: tagging is now possible on NetFlow v5-v8 engine_type/ engine_id, NetFlow v9/IPFIX source_id and sFlow AgentId. + tee plugin: added support for 'src_port' in tee_receivers map. When in non-transparent replication mode, use the specified UDP port to send data to receiver(s). This is in addition to tee_source_ip, which allows to set a configured IP address as source. + networks_no_mask_if_zero: a new knob so that IP prefixes with zero mask - that is, unknown ones or those hitting a default route - are not masked. The feature applies to *_net aggregation primitives and makes sure individual IP addresses belonging to unknown IP prefixes are not zeroed out. + networks_file: hooked up networks_file_no_lpm feature to peer and origin ASNs and (BGP) next-hop fields. + pmacctd: added support for calling pcap_set_protocol() if supported by libpcap. Patch is courtesy by Lennert Buytenhek ( @buytenh ). + pmbgpd, pmbmpd, pmtelemetryd: added a few CL options to ease output of BGP, BMP and Streaming Telemetry data, for example: -o supplies a b[gm]p_daemon_msglog_file, -O supplies a b[gm]p_dump_file and -i supplies b[gm]p_dump_refresh_time. + kafka plugin: in the examples section, added a Kafka consumer script using the performing confluent-kafka-python module. ! fix, BGP daemon: segfault with add-path enabled peers as per issue #128. Patch is courtesy by Markus Weber ( @FvDxxx ). ! fix, print plugin: do not update link to latest file if cause of purging is a safe action (ie. cache space is finished. Thanks to Camilo Cardona ( @jccardonar ) for reporting the issue. Also, for the same reason, do not execute triggers (ie. print_trigger_exec). ! fix, nfacctd: improved IP protocol check in NF_evaluate_flow_type() A missing length check was causing, under certain conditions, some flows to be marked as IPv6. Many thanks to Yann Belin for his support resolving the issue. ! fix, print and SQL plugins: optimized the cases when the dynamic filename/table has to be re-evaluated. This results in purge speed gains when the dynamic part is time-related and nfacctd_time_new is set to true. ! fix, bgp_daemon_md5_file: if the server socket is AF_INET and the compared peer address in MD5 file is AF_INET6 (v4-mapped v6), pass it through ipv4_mapped_to_ipv4(). Also if the server socket is AF_INET6 and the compared peer addess in MD5 file is AF_INET, pass it through ipv4_to_ipv4_mapped(). Thanks to Paul Mabey for reporting the issue. ! fix, nfacctd: improved length checks in resolve_vlen_template() to prevent SEGVs. Thanks to Josh Suhr and Levi Mason for their support. ! fix, nfacctd: flow stitching, improved flow end time checks. Thanks to Fabio Bindi ( @FabioLiv ) for his support resolving the issue. ! fix, amqp_common.c: amqp_persistent_msg now declares the RabbitMQ exchange as durable in addition to marking messages as persistent; this is related to issue #148. ! fix, nfacctd: added flowset count check to existing length checks for NetFlow v9/IPFIX datagrams. This is to avoid logs flooding in case of padding. Thanks to Steffen Plotner for reporting the issue. ! fix, BGP daemon: when dumping BGP data at regular time intervals, dump_close message contained wrongly formatted timestamp. Thanks to Yuri Lachin for reporting the issue. ! fix, MySQL plugin: if --enable-ipv6 and sql_num_hosts set to true, use INET6_ATON for both v4 and v6 addresses. Thanks to Guy Lowe ( @gunkaaa ) for reporting the issue and his support resolving it. ! fix, 'flows' primitive: it has been wired to sFlow so to count Flow Samples received. This is to support Q21 in FAQS document. ! fix, BGP daemon: Extended Communities value was printed with %d (signed) format string instead of %u (unsigned), causing issue on large values. ! fix, aggregate_primitives: improved support of 'u_int' semantics for 8 bytes integers. This is in addition to already supported 1, 2 and 4 bytes integers. ! fix, pidfile: pidfile created by plugin processes was not removed. Thanks to Yuri Lachin for reporting the issue. ! fix, print plugin: checking non-null file descriptor before setvbuf in order to prevent SEGV. Similar checks were added to prevent nulls be input to libavro calls when Apache Avro output is selected. ! fix, SQL plugins: MPLS aggregation primitives were not correctly activated in case sql_optimize_clauses was set to false. ! fix, building system: reviewed minimum requirement for libraries, removed unused m4 macros, split features in plugins (ie. MySQL) and supports (ie. JSON). ! fix, sql_history: it now correctly honors periods expressed is 's' seconds. ! fix, BGP daemon: rewritten bgp_peer_print() to be thread safe. ! fix, pretag.h: addressed compiler warning on 32-bit architectures, integer constant is too large for "long" type. Thanks to Stephen Clark ( @sclark46 ) for reporting the issue. - MongoDB plugin: it is being discontinued since the old Mongo API is not supported anymore and there has never been enough push from the community to transition to the new/current API (which would require a rewrite of most of the plugin). In this phase-1 the existing MongoDB plugin is still available using 'plugins: mongodb_legacy' in the configuration. - Packet classification basing on the L7-filter project is being discontinued (ie. 'classifiers' directive). This is being replaced by an implementation basing on the nDPI project. As part of this also the sql_aggressive_classification knob has been discontinued. - tee_receiver was part of the original implementation of the tee plugin, allowing to forward to a single target and hence requiring multiple plugins instantiated, one per target. Since 0.14.3 this directive was effectively outdated by tee_receivers. - tmp_net_own_field: the knob has been discontinued and was allowing to revert to backward compatible behaviour of IP prefixes (ie. src_net) being written in the same field as IP addresses (ie. src_host). - tmp_comms_same_field: the knob has been discontinued and was allowing to revert to backward compatible behaviour of BGP communities (standard and extended) being writeen all in the same field. - plugin_pipe_amqp and plugin_pipe_kafka features were meant as an alternative to the homegrown queue solution for internal messaging, ie. passing data from the Core Process to Plugins, and are being discontinued. They are being replaced by a new implementation, plugin_pipe_zmq, basing on ZeroMQ. - plugin_pipe_backlog was allowing to keep an artificial backlog of data in the Core Process so to maximise bypass poll() syscalls in plugins. If home-grown queueing is found limiting, instead of falling back to such strategies, ZeroMQ queueing should be used. - pmacctd: deprecated support for legacy link layers: FDDI, Token Ring and HDLC. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
