pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry.

A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP,
GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.



+ Introduced pmbgpd daemon: a stand-alone BGP collector daemon; acts as a
  passive neighbor and maintains per-peer RIBs; can log real-time and/or
  dump at regular time-intervals BGP data to configured backends.
+ Introduced pmbmpd daemon: a stand-alone BMP collector daemon; can log
  real-time and/or dump at regular time-intervals BMP and BGP data to
  configured backends.
+ Introduced Apache Avro as part of print, AMQP and Kafka output: Apache
  Avro is a data serialization system providing rich data structures, a
  compact, fast, binary data format, a container file to store persistent
  data, remote procedure call (RPC) and simple integration with dynamic
  languages. The implementation is courtesy by Codethink Ltd.
+ as_path, std_comm and ext_comm primitives: along with their src counter
  parts, ie. src_as_path etc., have been re-worked to a variagle-length
  internal representation which will lead, when using BGP primitives, to
  memory savings of up to 50% compared to previous releases. 
+ std_comm, ext_comm primitives: primitives are de-coupled so that they
  are not multiplexed anymore in the same field, on output. Added a
  tmp_comms_same_field config directive for backward compatibility.
+ nfacctd: added support for repeated NetFlow v9/IPFIX field types. Also
  flowStartDeltaMicroseconds (IE #158) and flowEndDeltaMicroseconds (#159)
  are now supported for timestamping. 
+ kafka plugin: it is now possible to specify -1 (RD_KAFKA_RTITION_UA) as
  part of the kafka_partition config directive. Also, introduced support
  for Kafka partition keys via kafka_partition_key and equivalent config
+ kafka plugin: kafka_broker_host directive now allows to specify multiple
  brokers, ie. "broker1:10000,broker2". The feature relies on capabilities
  of underlying rd_kafka_brokers_add().
+ tee, nfprobe, sfprobe plugins: introduced Kafka support for internal
  pipe and buffering, ie. plugin_pipe_kafka. This is in addition to the
  existing support for homegrown internal buffering and RabbitMQ.
+ tee plugin: introduced support for variable-length buffers which reduces
  CPU utilization.
+ print, MongoDB, AMQP and Kafka plugins: re-worked max_writers feature to
  not rely anymore on waitpid() inside signal handlers as it was failing on
  some OS versions (and could not be reproduced on others). Thanks to
  Janet Sullivan for her support.
+ bgp_follow_nexthop_external: introduced feature to return, when true, the
  next-hop from the routing table of the last node part of the supplied IP
  prefix(es) as value for the 'peer_ip_dst' primitive. When false, default,
  it returns the IP address of the last node part of the bgp_follow_nexthop 
  config key.
+ pmtelemetryd: added initial support for GPB. Input GPB data is currently
  base64'd in the telemetry_data field of the daemon output JSON object. 
+ pmtelemetryd: Added telemetry statistics. For each peer, track the number
  of packets received, how many bytes are pulled off the wire, and the
  resulting message payload. Dump these counts in logdump. Patch is courtesy
  by Tim LaBerge.
+ amqp_markers, kafka_markers: added start/end markers feature to AMQP and
  Kafka plugins output same as for the print plugin (print_markers).
+ pre_tag_map: 'direction' keyword now applies to sFlow too: it does expect
  values 0 (ingress direction) or 1 (egress direction), just like before.
  In sFlow v2/v4/v5 this returns a positive match if: 1) source_id equals
  to input interface and this 'direction' key is set to '0' or 2) source_id
  equals to output interface and this 'direction' key is set to '1'.
+ bgp_agent_map: introduced support for input and output interfaces. This
  is relevant to VPN scenarios.
+ tmp_asa_bi_flow hack: bi-flows use two counters to report counters, ie.
  bytes and packets, in forward and reverse directions. This hack (ab)uses
  the packets field in order to store the extra bytes counter.
! fix, nfacctd: debugging NetFlow v9/IPFIX templates, added original field
  type number to the output when the field is known and its description is
! fix, Jansson: added JSON_PRESERVE_ORDER flag to json_dumps() to give
  output consistency across runs.
! fix, kafka_common.c: added rd_kafka_message_destroy() to p_kafka_consume_
  _data() to prevent memory leaks. Thanks to Paul Mabey for his support
  solving the issue.
! fix, kafka_common.c: p_kafka_set_topic() now gives it some time for the
  topic to get (auto) created, if needed.
! fix, print plugin: improved check for when to print table title (csv,
  formatted). Either 1) print_output_file_append is set to false or 2)
  print_output_file_append is set to true and file is to be created.
! fix, print_markers: start marker is now printed also in the case where
  print_output_file_append is set to true. Also, markers are now printed as
  a JSON object, if output is set to JSON.
! fix, pkt_handlers.c: removed l3_proto checks from NF_peer_dst_ip_handler()
  for cases where a v6 flows has a v4 BGP next-hop (ie. vpnv6)
! fix, pre_tag_map: removed 32 chars length limit from set_label statement.
! fix, custom primitives: names are now interpreted as case-insensitive.
  Patch is courtesy by Corentin Neau.
! fix, BGP, BMP and Streaming Telemetry: if reopening [bgp, bmp, telemetry]_
  daemon_msglog_file via SIGHUP, reset reload flag.
! fix, BGP, BMP and Streaming Telemetry: removed gettimeofday() from bgp_
  peer_dump_init() and bgp_peer_dump_close() in order to maintain a single
  timestamp for a full dump event. Thanks to Tim LaBerge for his support.
! fix, BGP, BMP and Streaming Telemetry: output log and dump messages went
  through a general review to improve information consistency and usability.
  Message formats are now documented in docs/MSGLOG_DUMP_FORMATS so to more
  easily track future changes. 
! fix, pmtelemetryd: avoiding un-necessary spawn of a default plugin if none
  is defined.
! fix, pmtelemetryd: Mask SIGCHLD during socket IO. If we happen to be
  blocked in recv() while a log dump happens, recv() will fail with EINTR.
  This is to mask SIGCHLD during socket IO and restores the original mask
  after the IO completes. Patch is courtesy by Tim LaBerge.
! fix, build system: misc improvements made to the build system introduced
  in 1.6.0. Thanks to Vincent Bernat for his support in this area.
! fix, compiler warnings: ongoing effort to suppress warning messages when
  compiling. Thanks to Tim LaBerge, Matin Mitchell for their contributions.

See UPGRADE file.


pmacct-discussion mailing list

Reply via email to