VERSION. 1.7.2
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.2.tar.gz CHANGELOG. + nfacctd, sfacctd: added Kafka broker among the options to receive NetFlow/IPFIX, sFlow data from. Host, port and topic should all be specified along with an optional config file to pass to librdkafka. + nfacctd, sfacctd, pmtelemetryd: added ZeroMQ queue among the options to receive NetFlow/IPFIX, sFlow or Streaming Telemetry data from. An IP address and port should be specified. + nfacctd, sfacctd: added sampling_direction to the set of supported primitives, valid values being ingress, egress and unknown. + nfacctd, sfacctd: stats, ie. amount of NetFlow/IPFIX or sFlow packets received per router, are now available when in tee mode. Stats can be retrieved via a SIGUSR1 UNIX signal. + pcap_savefile_replay: a feature to replay content for the specified amounf of time when reading from a pcap_savefile. + pre_tag_map: added several new keys: src_net and dst_net (to tag on source and destination IP prefixes respectively), bgp_nexthop (to tag on BGP nexthop) and nat_event. + BGP daemon: added bgp_lrgcomm_pattern feature to filter large BGP communities (in addition to existing equivalent knobs to filter on standard and extended communities). + BMP, Streaming Telemetry daemons: msglog_file and dump_file config directives now offer $bmp_router, $bmp_router_port, $telemetry_node and $telemetry_node_port variables. + BGP, BMP, Streaming Telemetry daemons: added BGP, BMP and Streaming Telemetry exporter TCP/UDP port as variable for dump/log filenames (to better support NAT traversal scenarios). + BGP, BMP daemons: added message sequencing to both BGP and BMP dumps (bgp_table_dump_*, bmp_dump_*). If dumping and logging are enabled in parallel then sequencing the dumps allows for check pointing at regular time intervals. + BMP daemon: implemented draft-hsmit-bmp-extensible-routemon-msgs for a tlv-based encoding of route-monitoring messages with a new message type. + Streaming Telemetry daemon: added sample decoders for gRPC / GPB for Cisco and Huawei platforms, written in Python. Telemetry data is decoded using vendor-supplied proto files and output in JSON format in a ZeroMQ queue - suitable for ingestion in pmtelemetryd. Docs and sample code is available in the telemetry/ directory. This is all in addition to TCP/UDP transports and JSON encoding supported natively in pmtelemetryd. + kafka plugin: introduced support for Confluent Schema Registry via libserdes. A registry can be supplied via kafka_avro_schema_registry config directive; the schema is generated automatically. The feature enables validation of data passed through a Kafka broker and uses Avro encoding. + kafka plugin: added $in_iface key (input interface) to the set of variables supported by kafka_partition_key. Extremely useful when coupled to $peer_src_ip in some scenarios. + print, IMT plugins: separator for CSV format can now be space (\s) or tab (\t). + tee plugin: added Kafka broker among the emitters. kafka_broker and kafka_topic knobs are now available in the tee_receivers map and a tee_kafka_config_file directive allows to define a file with config to pass to librdkafka. + tee plugin: added ZeroMQ queue among the emitters. zmq_address knob defines the queue IP address and port to emit to. + tee plugin: introducing support for complex pre_tag_map when doing replication of NetFlow/IPFIX (sFlow replication had already this). With this feature flows are individually evaluated against supplied filters (input interface, BGP next-hop, etc.) and (not) replicated accordingly. + GeoIP v2: added support for latitude and longitude primitives via src_host_coords and dst_host_coords knobs. This is in addition to existing country and pocode supports. + files_uid, files_gid: now also user and group strings are accepted. This is in addition to user and group IDs. ! fix, nfacctd: NF_evaluate_flow_type() improved to not detect Cisco ASA flows (ie. those including initiator and responder octets) as events. Also improved sanity checking of received NetFlow v9/IPFIX data and options templates and reviwed modulo functions and improved template hashing. ! fix, BGP, BMP, Streaming Telemetry daemons: improved log sequencing by handling counter wrap-up more gracefully. Also a log sequencing API was developed to improve code re-use. ! fix, BGP daemon: added check for duplicate Router-IDs at BGP OPEN parsing time. If a duplicate is detected, the session BGP OPENing of the new session is dropped. ! fix, BGP daemon: ADD-PATH capability was checked only in the first AFI/SAFI and was being set in the reply for last AFI/SAFI RECEIVE(1) if first included SEND(2) or SEND-RECEIVE(3). Thanks to Markus Weber ( @FvDxxx ) for his patch. ! fix, BGP daemon: upon route lookup, don't perform ADD-PATH logics if no PATH-ID (even if ADD-PATH capability is announced by the peer). Thanks to Camilo Cardona ( @jccardonar ) for his support solving the issue. ! fix, BGP daemon: graceful handling of invalid AS-PATH segment types (ie. AS-PATH in BGP UPDATE inconsistent with capabilities passed in BGP OPEN) in order to avoid SEGVs. ! fix, pmtelemetryd: improved support for UDP timeouts. Also reviewed natively supported encodings: removed zjson and GPB was moved to pre- processors (with samples available in telemetry/decoders directory). ! fix, pmtelemetryd: no dump_init / dump_close events sequencing since all messages are sequenced anyway (consistency with other daemons). ! fix, kafka_common.c: now destroying both config and topic config as part of p_kafka_close() in order to avoid memory leaks. Also, port is omitted from broker string if not passed to p_kafka_set_broker(). And finally output queue length checks in p_kafka_check_outq_len() have been relaxed (to counter temporary hickups that need more patience). ! fix, kafka plugin: kafka_partition default was zero (that is, a valid partition number) instead of -1 (RD_KAFKA_PARTITION_UA or unassigned) which allows librdkafka to attach a partitioner. ! fix, SQL plugins: sql_table_schema is honoured even if sql_table_name is non-dynamic. This is to cover cases where the table is rotated externally. ! fix, mysql plugin: my_bool replaced with bool. The plugin now does compile against MySQL 8.0. Also added inclusion of stdbool.h as on some systems bool is not defined. Improved overall probing for MySQL headers. ! fix, pgsql plugin: sql_recovery_backup_host was not being honoured. PG_create_backend() now composes a proper conn_string. ! fix, print plugin: increase successful queries number, QN, only if the output file was successfully opened. ! fix, zmq_common.c: moved ZAP socket initialization inside the ZAP handler. See: https://github.com/zeromq/libzmq/issues/3313 . ! fix, util.c: length checks in handle_dynname_internal_strings() were reviewed. Existings were not working in absence of starting/trailing non-variable strings. ! fix, util.c: use lockf() instead of more problematic flock(). Thanks to Yuri Lachin ( @yuyutime ) and Miki Takata ( @mikiT ) for their support. ! fix, util.c: in compose_timestamp() pad usecs and use "%ld" since time fields are signed longs. Thanks to @raymondrussell for the patch. ! fix, ndpi_util.c: a protocol bitmask is now set in order to increase match rate. Patch is courtesy by @rsolsn. ! fix, compile time warnings: several warnings were addressed including but not restricted to -Wreturn-time, -Wunused-variable, implicit func declarations, -Wformat-extra-args, -Wunused-label, -Wunused-value, -Wunused-function, sbrk calls, -Wpointer-to-int-cast, -Wparentheses and -Wint-to-pointer-cast. ! fix, dangerous uninitialized values: net_aggr.c, pmacct.c: in merge() argument with non-NULL attribute could be passed NULL; bmp_msg.c: in bmp_process_msg_route_monitor() bdata.tstamp could be uninitialized; sfprobe_plugin.c: calloc() return value (possibly null) was not being checked; sflow_agent.c: uninitialized ret value in sfl_agent_init() could lead to undefined bind() error behaviour. ! fix, thread_pool.c: reviewed logics in deallocate_thread_pool() and solved a minor memory leak in allocate_thread_pool(). - pmacctd: removed support for FDDI :) - nfacctd: discontinued support for NetFlow v1, v7 and v8 collection and replication. - pre_tag_map: matching on 'sampling_rate' is not supported anymore as a sampling_rate primitive is now available; the 'return' feature to return matched data before completing the map workflow has started being obsoleted (retired from docs but still available). - plugin_pipe_check_core_pid: deprecating feature given RabbitMQ and Kafka are not supported anymore for internal message delivery. - tee plugin: obsoleted tee_dissect_send_full_pkt knob, entire packets are now replicated only if no pre_tag_map or a simple pre_tag_map is defined. - nfprobe plugin: removed support for NetFlow v1 export. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
