pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry. Each component
works both as a standalone daemon and as a thread of execution for correlation
purposes (ie. enrich NetFlow with BGP data).

A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to time-series databases like ElasticSearch and InfluxDB and
traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.



+ ZeroMQ integration: by defining plugin_pipe_zmq to 'true', ZeroMQ is
  used for queueing between the Core Process and plugins. This is in
  alternative to the home-grown circular queue implementation (ie.
  plugin_pipe_size). plugin_pipe_zmq_profile can be set to one value
  of { micro, small, medium, large, xlarge } and allows to select
  among a few standard buffering profiles without having to fiddle
  with plugin_buffer_size. How to compile, install and operate ZeroMQ
  is documented in the "Internal buffering and queueing" section of
  the QUICKSTART document. 
+ nDPI integration: enables packet classification, replacing existing
  L7-layer project integration, and is available for pmacctd and
  uacctd. The feature, once nDPI is compiled in, is simply enabled by
  specifying 'class' as part of the aggregation method. How to compile
  install and operate nDPI is documented in the "Quickstart guide to
  packet classification" section of the QUICKSTART document.
+ nfacctd: introduced nfacctd_templates_file so that NetFlow v9/IPFIX
  templates can be cached to disk to limit the amount of lost packets
  due to unknown templates when nfacctd (re)starts. The implementation
  is courtesy by Codethink Ltd. 
+ nfacctd: introduced support for PEN on IPFIX option templates. This
  is in addition to already supported PEN for data templates. Thanks
  to Gilad Zamoshinski ( @zamog ) for his support. 
+ sfacctd: introduced new aggregation primitives (tunnel_src_host,
  tunnel_dst_host, tunnel_proto, tunnel_tos) to support inner L3
  layers. Thanks to Kaname Nishizuka ( @__kaname__ ) for his support.
+ nfacctd, sfacctd: pcap_savefile and pcap_savefile_wait were ported
  from pmacctd. They allow to process NetFlow/IPFIX and sFlow data
  from previously captured packets; these also ease some debugging by
  not having to resort anymore to tcpreplay for most cases.
+ pmacctd, sfacctd: nfacctd_time_new feature has been ported so, when
  historical accounting is enabled, to allow to choose among capture
  time and time of receipt at the collector for time-binning.
+ nfacctd: added support for NetFlow v9/IPFIX field types #130/#131,
  respectively the IPv4/IPv6 address of the element exporter.
+ nfacctd: introduced nfacctd_disable_opt_scope_check: mainly a work
  around to implementations not encoding NetFlow v9/IPIFX option scope
  correctly, this knob allows to disable option scope checking. Thanks
  to Gilad Zamoshinski ( @zamog ) for his support.
+ pre_tag_map: added 'source_id' key for tagging on NetFlow v9/IPFIX
  source_id field. Added also 'fwdstatus' for tagging on NetFlow v9/
  IPFIX information element #89: this implementation is courtesy by
  Emil Palm ( @mrevilme ).
+ tee plugin: tagging is now possible on NetFlow v5-v8 engine_type/
  engine_id, NetFlow v9/IPFIX source_id and sFlow AgentId.
+ tee plugin: added support for 'src_port' in tee_receivers map. When
  in non-transparent replication mode, use the specified UDP port to
  send data to receiver(s). This is in addition to tee_source_ip,
  which allows to set a configured IP address as source.
+ networks_no_mask_if_zero: a new knob so that IP prefixes with zero
  mask - that is, unknown ones or those hitting a default route - are
  not masked. The feature applies to *_net aggregation primitives and
  makes sure individual IP addresses belonging to unknown IP prefixes
  are not zeroed out.
+ networks_file: hooked up networks_file_no_lpm feature to peer and
  origin ASNs and (BGP) next-hop fields.
+ pmacctd: added support for calling pcap_set_protocol() if supported
  by libpcap. Patch is courtesy by Lennert Buytenhek ( @buytenh ).
+ pmbgpd, pmbmpd, pmtelemetryd: added a few CL options to ease output
  of BGP, BMP and Streaming Telemetry data, for example: -o supplies
  a b[gm]p_daemon_msglog_file, -O supplies a b[gm]p_dump_file and -i
  supplies b[gm]p_dump_refresh_time.
+ kafka plugin: in the examples section, added a Kafka consumer script
  using the performing confluent-kafka-python module. 
! fix, BGP daemon: segfault with add-path enabled peers as per issue
  #128. Patch is courtesy by Markus Weber ( @FvDxxx ).
! fix, print plugin: do not update link to latest file if cause of
  purging is a safe action (ie. cache space is finished. Thanks to
  Camilo Cardona ( @jccardonar ) for reporting the issue. Also, for
  the same reason, do not execute triggers (ie. print_trigger_exec). 
! fix, nfacctd: improved IP protocol check in NF_evaluate_flow_type()
  A missing length check was causing, under certain conditions, some
  flows to be marked as IPv6. Many thanks to Yann Belin for his
  support resolving the issue.
! fix, print and SQL plugins: optimized the cases when the dynamic
  filename/table has to be re-evaluated. This results in purge speed
  gains when the dynamic part is time-related and nfacctd_time_new is
  set to true.
! fix, bgp_daemon_md5_file: if the server socket is AF_INET and the
  compared peer address in MD5 file is AF_INET6 (v4-mapped v6), pass
  it through ipv4_mapped_to_ipv4(). Also if the server socket is
  AF_INET6 and the compared peer addess in MD5 file is AF_INET, pass
  it through ipv4_to_ipv4_mapped(). Thanks to Paul Mabey for reporting
  the issue.
! fix, nfacctd: improved length checks in resolve_vlen_template() to
  prevent SEGVs. Thanks to Josh Suhr and Levi Mason for their support.
! fix, nfacctd: flow stitching, improved flow end time checks. Thanks
  to Fabio Bindi ( @FabioLiv ) for his support resolving the issue.
! fix, amqp_common.c: amqp_persistent_msg now declares the RabbitMQ
  exchange as durable in addition to marking messages as persistent;
  this is related to issue #148.
! fix, nfacctd: added flowset count check to existing length checks 
  for NetFlow v9/IPFIX datagrams. This is to avoid logs flooding in
  case of padding. Thanks to Steffen Plotner for reporting the issue.
! fix, BGP daemon: when dumping BGP data at regular time intervals,
  dump_close message contained wrongly formatted timestamp. Thanks to
  Yuri Lachin for reporting the issue.
! fix, MySQL plugin: if --enable-ipv6 and sql_num_hosts set to true,
  use INET6_ATON for both v4 and v6 addresses. Thanks to Guy Lowe
  ( @gunkaaa ) for reporting the issue and his support resolving it.
! fix, 'flows' primitive: it has been wired to sFlow so to count Flow
  Samples received. This is to support Q21 in FAQS document.
! fix, BGP daemon: Extended Communities value was printed with %d
  (signed) format string instead of %u (unsigned), causing issue on
  large values.
! fix, aggregate_primitives: improved support of 'u_int' semantics for
  8 bytes integers. This is in addition to already supported 1, 2 and
  4 bytes integers.
! fix, pidfile: pidfile created by plugin processes was not removed.
  Thanks to Yuri Lachin for reporting the issue.
! fix, print plugin: checking non-null file descriptor before setvbuf
  in order to prevent SEGV. Similar checks were added to prevent nulls
  be input to libavro calls when Apache Avro output is selected.
! fix, SQL plugins: MPLS aggregation primitives were not correctly
  activated in case sql_optimize_clauses was set to false.
! fix, building system: reviewed minimum requirement for libraries,
  removed unused m4 macros, split features in plugins (ie. MySQL) and
  supports (ie. JSON).
! fix, sql_history: it now correctly honors periods expressed is 's'
! fix, BGP daemon: rewritten bgp_peer_print() to be thread safe.
! fix, pretag.h: addressed compiler warning on 32-bit architectures,
  integer constant is too large for "long" type. Thanks to Stephen
  Clark ( @sclark46 ) for reporting the issue.
- MongoDB plugin: it is being discontinued since the old Mongo API is
  not supported anymore and there has never been enough push from the
  community to transition to the new/current API (which would require
  a rewrite of most of the plugin). In this phase-1 the existing
  MongoDB plugin is still available using 'plugins: mongodb_legacy'
  in the configuration.
- Packet classification basing on the L7-filter project is being
  discontinued (ie. 'classifiers' directive). This is being replaced
  by an implementation basing on the nDPI project. As part of this
  also the sql_aggressive_classification knob has been discontinued.
- tee_receiver was part of the original implementation of the tee
  plugin, allowing to forward to a single target and hence requiring
  multiple plugins instantiated, one per target. Since 0.14.3 this
  directive was effectively outdated by tee_receivers.
- tmp_net_own_field: the knob has been discontinued and was allowing
  to revert to backward compatible behaviour of IP prefixes (ie.
  src_net) being written in the same field as IP addresses (ie.
- tmp_comms_same_field: the knob has been discontinued and was
  allowing to revert to backward compatible behaviour of BGP
  communities (standard and extended) being writeen all in the same
- plugin_pipe_amqp and plugin_pipe_kafka features were meant as an
  alternative to the homegrown queue solution for internal messaging,
  ie. passing data from the Core Process to Plugins, and are being
  discontinued. They are being replaced by a new implementation,
  plugin_pipe_zmq, basing on ZeroMQ.
- plugin_pipe_backlog was allowing to keep an artificial backlog of
  data in the Core Process so to maximise bypass poll() syscalls in
  plugins. If home-grown queueing is found limiting, instead of
  falling back to such strategies, ZeroMQ queueing should be used.
- pmacctd: deprecated support for legacy link layers: FDDI, Token Ring
  and HDLC.

See UPGRADE file.


pmacct-discussion mailing list

Reply via email to