VERSION. 1.6.2
DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.6.2.tar.gz CHANGELOG. + BGP, BMP daemons: introduced support for BGP Large Communities IETF draft (draft-ietf-idr-large-community). Large Communities are stored in a variable-length field. Thanks to Job Snijders ( @job ) for his support. + BGP daemon: implemented draft-ietf-idr-shutdown. The draft defines a mechanism to transmit a short freeform UTF-8 message as part of a Cease NOTIFICATION message to inform the peer why the BGP session is being shutdown or reset. Thanks to Job Snijders ( @job ) for his support. + tee plugin, pre_tag_map: introduced support for inspetion of specific flow primitives and selective replication over them. The primitives supported are: input and output interfaces, source and destination MAC addresses, VLAN ID. The feature is now limited to sFlow v5 only. Thanks to Nick Hilliard and Barry O'Donovan for their support. + Added src_host_pocode and dst_host_pocode primitives, pocode being a compact and (de-)aggregatable (easy to identify districts, cities, metro areas, etc.) geographical representation, based on the Maxmind v2 City Database. Thanks to Jerred Horsman for his support. + Kafka support: introduced support for user-defined (librdkafka) config file via the new *_kafka_config_file config directives. Full pathname to a file containing directives to configure librdkafka is expected. All knobs whose values are string, integer, boolean are supported. + AMQP, Kafka plugins: introduced new directives kafka_avro_schema_topic, amqp_avro_schema_routing_key to transmit Apache Avro schemas at regular time intervals. The routing key/topic can overlap with the one used to send actual data. + AMQP, Kafka plugins: introduced support for start/stop markers when encoding is set to Avro (ie. 'kafka_output: avro'); also Avro schema is now embedded in a JSON envelope when sending it via a topic/routing key (ie. kafka_avro_schema_topic). + print plugin: introduced new config directive avro_schema_output_file to save the Apache Avro schema in a separate file (it was only possible to have it combined at the beginning of the data file). + BGP daemon: introduced a new bgp_daemon_as config directive to set a LocalAS which could be different from the remote peer one. This is to establish an eBGP session instead of a iBGP one (default). + flow_to_rd_map: introduced support for mpls_vpn_id. In NetFlow/IPFIX this is compared against Field Types #234 and #235. + sfacctd: introduced support for sFlow v2/v4 counter samples (generic, ethernet, vlan). This is in addition to existing support for sFlow v5 counters. + BGP, BMP and Streming Telemetry daemons: added writer_id field when writing to Kafka and/or RabbitMQ. The field reports the configured core_proc_name and the actual PID of the writer process (so, while being able to correlate writes to the same daemon, it's also possible to distinguish among overlapping writes). + amqp, kafka, print plugins: harmonized JSON output to the above: added event_type field, writer_id field with plugin name and PID. + BGP, BMP daemons: added AFI, SAFI information to log and dump outputs; also show VPN Label if SAFI is MPLS VPN. + pmbgpd, pmbmpd: added logics to bypass building RIBs if only logging BGP/BMP data real-time. + BMP daemon: added BMP peer TCP port to log and dump outputs (for NAT traversal scenarios). Contextually, multiple TCP sessions per IP are now supported for the same reason. + SQL plugins: ported (from print, etc. plugins) the 1.6.1 re-working of the max_writers feature. + uacctd: use current time when we don't have a timestamp from netlink. We only get a timestamp when there is a timestamp in the skb. Notably, locally generated packets don't get a timestamp. The patch is courtesy by Vincent Bernat ( @vincentbernat ). + build system: added configure options for partial linking of binaries with any selection/combination of IPv4/IPv6 accounting daemons, BGP daemon, BMP daemon and Streaming Telemetry daemon possible. By default all are compiled in. + BMP daemon: internal code changes to pass additional info from BMP per-peer header to bgp_parse_update_msg(). Goal is to expose further info, ie. pre- vs post- policy, when logging or dumping BMP info. ! fix, BGP daemon: introduced parsing of IPv6 MPLS VPN (vpnv6) NLRIs. Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue. ! fix, BGP daemon: upon doing routes lookup, now correctly honouring the case of BGP-LU (SAFI_MPLS_LABEL). ! fix, BGP daemon: send BGP NOTIFICATION out in case of known failures in bgp_parse_msg(). ! fix, kafka_partition, *_kafka_partition: default value changed from 0 (partition zero) to -1 (RD_KAFKA_PARTITION_UA, partition unassigned). Thanks to Johan van den Dorpe ( @johanek ) for his support. ! fix, pre_tag_map: removed constraint for 'ip' keyword for nfacctd and sfacctd maps. While this is equivalent syntax to specifying rules with 'ip=0.0.0.0/0', it allows for map indexing (maps_index: true). ! fix, bgp_agent_map: improved sanity check against bgp_ip for IPv6 addresses (ie. an issue appeared for the case of '::1' where the first 64 bits are zeroed out). Thanks to Charlie Smurthwaite ( @catphish ) for reporting the issue. ! fix, maps_index: indexing now correctly works for IPv6 pre_tag_map entries. That is, those where 'ip', the IP address of the NetFlow/ IPFIX/sFlow exporter, is an IPv6 address. ! fix, pre_tag_map: if mpls_vpn_rd matching condition is specified and maps_index is enabled, PT_map_index_fdata_mpls_vpn_rd_handler() now picks the right (and expected) info. ! fix, pkt_handlers.c: improved definition and condition to free() in bgp_ext_handler() in order to prevent SEGVs. Thanks to Paul Mabey for his support. ! fix, kafka_common.c: removed waiting time from p_kafka_set_topic(). Added docs advicing to create in advance Kafka topics. ! fix, sfacctd, sfprobe: tag and tag2 are now correctly re-defined as 64 bits long. ! fix, sfprobe plugin, sfacctd: tags and class primitives are now being encoded/decoded using enterprise #43874, legit, instead of #8800, that was squatted back in the times. See issue #71 on GiHub for more info. ! fix, sfacctd: lengthCheck() + skipBytes() were producing an incorrect jump in case of unknown flow samples. Replaced by skipBytesAndCheck(). Thanks to Elisa Jasinska ( @fooelisa ) for her support. ! fix, pretag_handlers.c: in bgp_agent_map added case for 'vlan and ...' filter values. ! fix, BGP daemon: multiple issues of partial visibility of the stored RIBs and SEGVs when bgp_table_per_peer_buckets was not left default: don't mess with bms->table_per_peer_buckets given the multi-threaded scenario. Thanks to Dan Berger ( @dfberger ) for his support. ! fix, BGP, BMP daemons: bgp_process_withdraw() function init aligned to bgp_process_update() in order to prevent SEGVs. Thanks to Yuri Lachin for his support. ! fix, bgp_msg.c: Route Distinguisher was stored and printed incorrectly when of type RD_TYPE_IP. Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue. ! fix, bgp_logdump.c: p_kafka_set_topic() was being wrongly applied to an amqp_host structure (instead of a kafka_host structure). Thanks to Corentin Neau ( @weyfonk ) for reporting the issue. ! fix, BGP daemon: improved BGP next-hop setting and comparison in cases of MP_REACH_NLRI and MPLS VPNs. Many thanks to both Catalin Petrescu ( @cpmarvin ) and Alberto Santos ( @m4ccbr ) for their support. ! fix, pmbgpd, pmbmpd: pidfile was not written even if configured. Thanks to Aaron Glenn ( @aaglenn ) for reporting the issue. ! fix, tee plugin: tee_max_receiver_pools is now correctly honoured and debug message shows the replicatd protocol, ie. NetFlow/IPFIX vs sFlow. ! AMQP, Kafka plugins: separate JSON objects, newline separated, are preferred to JSON arrays when buffering of output is enabled (ie. kafka_multi_values) and output is set to JSON. This is due to quicker serialisation performance shown by the Jansson library. ! build system: switched to enable IPv6 support by default (while the --disable-ipv6 knob can be used to reverse the behaviour). Patch is courtesy by Elisa Jasinska ( @fooelisa ). ! build system: given visibility, ie. via -V CL option, into compile options enabled by default (ie. IPv6, threads, 64bit counters, etc.). ! fix, nfprobe: free expired records when exporting to an unavailable collector in order to prevent a memory leak. Patch is courtersy by Vladimir Kunschikov ( @kunschikov ). ! fix, AMQP plugin: set content type to binary in case of Apache Avro output. ! fix, AMQP, Kafka plugins: optimized amqp_avro_schema_routing_key and kafka_avro_schema_topic. Avro schema is built only once at startup. ! fix, cfg.c: improved parsing of config key-values where squared brakets appear in the value part. Thanks to Brad Hein ( @regulatre ) for reporting the issue. Also, detection of duplicates among plugin and core process names was improved. ! fix, misc: compiler warnings: fix up missing includes and prototypes; the patch is courtesy by Tim LaBerge ( @tlaberge ). ! kafka_consumer.py, amqp_receiver.py: Kafka, RabbitMQ consumer example scripts have been greatly expanded to support posting to a REST API or to a new Kafka topic, including some stats. Also conversion of multiple newline-separated JSON objects to a JSON array has been added. Misc bugs were fixed. NOTES. See UPGRADE file. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
