pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry. Each component
works both as a standalone daemon and as a thread of execution for correlation
purposes (ie. enrich NetFlow with BGP data).

A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP,
GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.



+ BGP, BMP daemons: introduced support for BGP Large Communities IETF
  draft (draft-ietf-idr-large-community). Large Communities are stored
  in a variable-length field. Thanks to Job Snijders ( @job ) for his
+ BGP daemon: implemented draft-ietf-idr-shutdown. The draft defines a
  mechanism to transmit a short freeform UTF-8 message as part of a
  Cease NOTIFICATION message to inform the peer why the BGP session is
  being shutdown or reset. Thanks to Job Snijders ( @job ) for his
+ tee plugin, pre_tag_map: introduced support for inspetion of specific
  flow primitives and selective replication over them. The primitives
  supported are: input and output interfaces, source and destination
  MAC addresses, VLAN ID. The feature is now limited to sFlow v5 only. 
  Thanks to Nick Hilliard and Barry O'Donovan for their support.
+ Added src_host_pocode and dst_host_pocode primitives, pocode being a
  compact and (de-)aggregatable (easy to identify districts, cities,
  metro areas, etc.) geographical representation, based on the Maxmind
  v2 City Database. Thanks to Jerred Horsman for his support.
+ Kafka support: introduced support for user-defined (librdkafka) config
  file via the new *_kafka_config_file config directives. Full pathname
  to a file containing directives to configure librdkafka is expected.
  All knobs whose values are string, integer, boolean are supported.
+ AMQP, Kafka plugins: introduced new directives kafka_avro_schema_topic,
  amqp_avro_schema_routing_key to transmit Apache Avro schemas at regular
  time intervals. The routing key/topic can overlap with the one used to
  send actual data.
+ AMQP, Kafka plugins: introduced support for start/stop markers when
  encoding is set to Avro (ie. 'kafka_output: avro'); also Avro schema
  is now embedded in a JSON envelope when sending it via a topic/routing
  key (ie. kafka_avro_schema_topic).
+ print plugin: introduced new config directive avro_schema_output_file
  to save the Apache Avro schema in a separate file (it was only possible
  to have it combined at the beginning of the data file).  
+ BGP daemon: introduced a new bgp_daemon_as config directive to set a
  LocalAS which could be different from the remote peer one. This is to
  establish an eBGP session instead of a iBGP one (default).
+ flow_to_rd_map: introduced support for mpls_vpn_id. In NetFlow/IPFIX
  this is compared against Field Types #234 and #235.
+ sfacctd: introduced support for sFlow v2/v4 counter samples (generic,
  ethernet, vlan). This is in addition to existing support for sFlow v5
+ BGP, BMP and Streming Telemetry daemons: added writer_id field when
  writing to Kafka and/or RabbitMQ. The field reports the configured
  core_proc_name and the actual PID of the writer process (so, while
  being able to correlate writes to the same daemon, it's also possible
  to distinguish among overlapping writes).
+ amqp, kafka, print plugins: harmonized JSON output to the above: added
  event_type field, writer_id field with plugin name and PID.
+ BGP, BMP daemons: added AFI, SAFI information to log and dump outputs;
  also show VPN Label if SAFI is MPLS VPN.
+ pmbgpd, pmbmpd: added logics to bypass building RIBs if only logging
  BGP/BMP data real-time.
+ BMP daemon: added BMP peer TCP port to log and dump outputs (for NAT
  traversal scenarios). Contextually, multiple TCP sessions per IP are
  now supported for the same reason.
+ SQL plugins: ported (from print, etc. plugins) the 1.6.1 re-working of
  the max_writers feature.
+ uacctd: use current time when we don't have a timestamp from netlink.
  We only get a timestamp when there is a timestamp in the skb. Notably,
  locally generated packets don't get a timestamp. The patch is courtesy
  by Vincent Bernat ( @vincentbernat ).
+ build system: added configure options for partial linking of binaries
  with any selection/combination of IPv4/IPv6 accounting daemons, BGP
  daemon, BMP daemon and Streaming Telemetry daemon possible. By default
  all are compiled in.
+ BMP daemon: internal code changes to pass additional info from BMP
  per-peer header to bgp_parse_update_msg(). Goal is to expose further
  info, ie. pre- vs post- policy, when logging or dumping BMP info.
! fix, BGP daemon: introduced parsing of IPv6 MPLS VPN (vpnv6) NLRIs.
  Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue.
! fix, BGP daemon: upon doing routes lookup, now correctly honouring
  the case of BGP-LU (SAFI_MPLS_LABEL). 
! fix, BGP daemon: send BGP NOTIFICATION out in case of known failures
  in bgp_parse_msg().
! fix, kafka_partition, *_kafka_partition: default value changed from 0
  (partition zero) to -1 (RD_KAFKA_PARTITION_UA, partition unassigned).
  Thanks to Johan van den Dorpe ( @johanek ) for his support.
! fix, pre_tag_map: removed constraint for 'ip' keyword for nfacctd and
  sfacctd maps. While this is equivalent syntax to specifying rules with
  'ip=', it allows for map indexing (maps_index: true).
! fix, bgp_agent_map: improved sanity check against bgp_ip for IPv6
  addresses (ie. an issue appeared for the case of '::1' where the first
  64 bits are zeroed out). Thanks to Charlie Smurthwaite ( @catphish )
  for reporting the issue.
! fix, maps_index: indexing now correctly works for IPv6 pre_tag_map
  entries. That is, those where 'ip', the IP address of the NetFlow/
  IPFIX/sFlow exporter, is an IPv6 address.
! fix, pre_tag_map: if mpls_vpn_rd matching condition is specified and
  maps_index is enabled, PT_map_index_fdata_mpls_vpn_rd_handler() now
  picks the right (and expected) info.
! fix, pkt_handlers.c: improved definition and condition to free() in
  bgp_ext_handler() in order to prevent SEGVs. Thanks to Paul Mabey for
  his support.
! fix, kafka_common.c: removed waiting time from p_kafka_set_topic().
  Added docs advicing to create in advance Kafka topics.
! fix, sfacctd, sfprobe: tag and tag2 are now correctly re-defined as
  64 bits long.
! fix, sfprobe plugin, sfacctd: tags and class primitives are now being
  encoded/decoded using enterprise #43874, legit, instead of #8800, that
  was squatted back in the times. See issue #71 on GiHub for more info.
! fix, sfacctd: lengthCheck() + skipBytes() were producing an incorrect
  jump in case of unknown flow samples. Replaced by skipBytesAndCheck().
  Thanks to Elisa Jasinska ( @fooelisa ) for her support.
! fix, pretag_handlers.c: in bgp_agent_map added case for 'vlan and ...'
  filter values.
! fix, BGP daemon: multiple issues of partial visibility of the stored
  RIBs and SEGVs when bgp_table_per_peer_buckets was not left default:
  don't mess with bms->table_per_peer_buckets given the multi-threaded
  scenario. Thanks to Dan Berger ( @dfberger ) for his support.
! fix, BGP, BMP daemons: bgp_process_withdraw() function init aligned to
  bgp_process_update() in order to prevent SEGVs. Thanks to Yuri Lachin
  for his support.
! fix, bgp_msg.c: Route Distinguisher was stored and printed incorrectly
  when of type RD_TYPE_IP. Thanks to Alberto Santos ( @m4ccbr ) for
  reporting the issue.
! fix, bgp_logdump.c: p_kafka_set_topic() was being wrongly applied to
  an amqp_host structure (instead of a kafka_host structure). Thanks to
  Corentin Neau ( @weyfonk ) for reporting the issue.
! fix, BGP daemon: improved BGP next-hop setting and comparison in cases
  of MP_REACH_NLRI and MPLS VPNs. Many thanks to both Catalin Petrescu
  ( @cpmarvin ) and Alberto Santos ( @m4ccbr ) for their support.
! fix, pmbgpd, pmbmpd: pidfile was not written even if configured. Thanks
  to Aaron Glenn ( @aaglenn ) for reporting the issue.
! fix, tee plugin: tee_max_receiver_pools is now correctly honoured and
  debug message shows the replicatd protocol, ie. NetFlow/IPFIX vs sFlow.
! AMQP, Kafka plugins: separate JSON objects, newline separated, are
  preferred to JSON arrays when buffering of output is enabled (ie.
  kafka_multi_values) and output is set to JSON. This is due to quicker
  serialisation performance shown by the Jansson library.
! build system: switched to enable IPv6 support by default (while the
  --disable-ipv6 knob can be used to reverse the behaviour). Patch is
  courtesy by Elisa Jasinska ( @fooelisa ).
! build system: given visibility, ie. via -V CL option, into compile
  options enabled by default (ie. IPv6, threads, 64bit counters, etc.).
! fix, nfprobe: free expired records when exporting to an unavailable
  collector in order to prevent a memory leak. Patch is courtersy by
  Vladimir Kunschikov ( @kunschikov ).
! fix, AMQP plugin: set content type to binary in case of Apache Avro
! fix, AMQP, Kafka plugins: optimized amqp_avro_schema_routing_key and
  kafka_avro_schema_topic. Avro schema is built only once at startup.
! fix, cfg.c: improved parsing of config key-values where squared brakets
  appear in the value part. Thanks to Brad Hein ( @regulatre ) for
  reporting the issue. Also, detection of duplicates among plugin and
  core process names was improved. 
! fix, misc: compiler warnings: fix up missing includes and prototypes;
  the patch is courtesy by Tim LaBerge ( @tlaberge ).
!, Kafka, RabbitMQ consumer example
  scripts have been greatly expanded to support posting to a REST API or
  to a new Kafka topic, including some stats. Also conversion of multiple
  newline-separated JSON objects to a JSON array has been added. Misc
  bugs were fixed.

See UPGRADE file.


pmacct-discussion mailing list

Reply via email to