VERSION.
1.7.2


DESCRIPTION.
pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry. Each component
works both as a standalone daemon and as a thread of execution for correlation
purposes (ie. enrich NetFlow with BGP data).

A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to time-series databases like ElasticSearch and InfluxDB and
traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.


HOMEPAGE.
http://www.pmacct.net/


DOWNLOAD.
http://www.pmacct.net/pmacct-1.7.2.tar.gz


CHANGELOG.
+ nfacctd, sfacctd: added Kafka broker among the options to receive
  NetFlow/IPFIX, sFlow data from. Host, port and topic should all be
  specified along with an optional config file to pass to librdkafka.
+ nfacctd, sfacctd, pmtelemetryd: added ZeroMQ queue among the options
  to receive NetFlow/IPFIX, sFlow or Streaming Telemetry data from. An
  IP address and port should be specified.
+ nfacctd, sfacctd: added sampling_direction to the set of supported
  primitives, valid values being ingress, egress and unknown.
+ nfacctd, sfacctd: stats, ie. amount of NetFlow/IPFIX or sFlow packets
  received per router, are now available when in tee mode. Stats can be
  retrieved via a SIGUSR1 UNIX signal.
+ pcap_savefile_replay: a feature to replay content for the specified
  amounf of time when reading from a pcap_savefile.
+ pre_tag_map: added several new keys: src_net and dst_net (to tag on
  source and destination IP prefixes respectively), bgp_nexthop (to
  tag on BGP nexthop) and nat_event.
+ BGP daemon: added bgp_lrgcomm_pattern feature to filter large BGP
  communities (in addition to existing equivalent knobs to filter on
  standard and extended communities).
+ BMP, Streaming Telemetry daemons: msglog_file and dump_file config
  directives now offer $bmp_router, $bmp_router_port, $telemetry_node
  and $telemetry_node_port variables.
+ BGP, BMP, Streaming Telemetry daemons: added BGP, BMP and Streaming
  Telemetry exporter TCP/UDP port as variable for dump/log filenames
  (to better support NAT traversal scenarios).
+ BGP, BMP daemons: added message sequencing to both BGP and BMP dumps
  (bgp_table_dump_*, bmp_dump_*). If dumping and logging are enabled
  in parallel then sequencing the dumps allows for check pointing at
  regular time intervals.
+ BMP daemon: implemented draft-hsmit-bmp-extensible-routemon-msgs for
  a tlv-based encoding of route-monitoring messages with a new message
  type.  
+ Streaming Telemetry daemon: added sample decoders for gRPC / GPB for
  Cisco and Huawei platforms, written in Python. Telemetry data is
  decoded using vendor-supplied proto files and output in JSON format
  in a ZeroMQ queue - suitable for ingestion in pmtelemetryd. Docs and
  sample code is available in the telemetry/ directory. This is all in
  addition to TCP/UDP transports and JSON encoding supported natively
  in pmtelemetryd.
+ kafka plugin: introduced support for Confluent Schema Registry via
  libserdes. A registry can be supplied via kafka_avro_schema_registry
  config directive; the schema is generated automatically. The feature
  enables validation of data passed through a Kafka broker and uses
  Avro encoding. 
+ kafka plugin: added $in_iface key (input interface) to the set of
  variables supported by kafka_partition_key. Extremely useful when
  coupled to $peer_src_ip in some scenarios. 
+ print, IMT plugins: separator for CSV format can now be space (\s)
  or tab (\t).
+ tee plugin: added Kafka broker among the emitters. kafka_broker and
  kafka_topic knobs are now available in the tee_receivers map and a
  tee_kafka_config_file directive allows to define a file with config
  to pass to librdkafka.
+ tee plugin: added ZeroMQ queue among the emitters. zmq_address knob
  defines the queue IP address and port to emit to.
+ tee plugin: introducing support for complex pre_tag_map when doing
  replication of NetFlow/IPFIX (sFlow replication had already this).
  With this feature flows are individually evaluated against supplied
  filters (input interface, BGP next-hop, etc.) and (not) replicated
  accordingly.
+ GeoIP v2: added support for latitude and longitude primitives via
  src_host_coords and dst_host_coords knobs. This is in addition to
  existing country and pocode supports.
+ files_uid, files_gid: now also user and group strings are accepted.
  This is in addition to user and group IDs.
! fix, nfacctd: NF_evaluate_flow_type() improved to not detect Cisco
  ASA flows (ie. those including initiator and responder octets) as
  events. Also improved sanity checking of received NetFlow v9/IPFIX
  data and options templates and reviwed modulo functions and improved
  template hashing.
! fix, BGP, BMP, Streaming Telemetry daemons: improved log sequencing
  by handling counter wrap-up more gracefully. Also a log sequencing
  API was developed to improve code re-use. 
! fix, BGP daemon: added check for duplicate Router-IDs at BGP OPEN
  parsing time. If a duplicate is detected, the session BGP OPENing of
  the new session is dropped.
! fix, BGP daemon: ADD-PATH capability was checked only in the first
  AFI/SAFI and was being set in the reply for last AFI/SAFI RECEIVE(1)
  if first included SEND(2) or SEND-RECEIVE(3). Thanks to Markus Weber
  ( @FvDxxx ) for his patch.
! fix, BGP daemon: upon route lookup, don't perform ADD-PATH logics if
  no PATH-ID (even if ADD-PATH capability is announced by the peer).
  Thanks to Camilo Cardona ( @jccardonar ) for his support solving the
  issue.
! fix, BGP daemon: graceful handling of invalid AS-PATH segment types
  (ie. AS-PATH in BGP UPDATE inconsistent with capabilities passed in
  BGP OPEN) in order to avoid SEGVs.
! fix, pmtelemetryd: improved support for UDP timeouts. Also reviewed
  natively supported encodings: removed zjson and GPB was moved to pre-
  processors (with samples available in telemetry/decoders directory). 
! fix, pmtelemetryd: no dump_init / dump_close events sequencing since
  all messages are sequenced anyway (consistency with other daemons).
! fix, kafka_common.c: now destroying both config and topic config as
  part of p_kafka_close() in order to avoid memory leaks. Also, port is
  omitted from broker string if not passed to p_kafka_set_broker(). And
  finally output queue length checks in p_kafka_check_outq_len() have
  been relaxed (to counter temporary hickups that need more patience). 
! fix, kafka plugin: kafka_partition default was zero (that is, a valid
  partition number) instead of -1 (RD_KAFKA_PARTITION_UA or unassigned)
  which allows librdkafka to attach a partitioner.
! fix, SQL plugins: sql_table_schema is honoured even if sql_table_name
  is non-dynamic. This is to cover cases where the table is rotated
  externally.
! fix, mysql plugin: my_bool replaced with bool. The plugin now does
  compile against MySQL 8.0. Also added inclusion of stdbool.h as on
  some systems bool is not defined. Improved overall probing for MySQL
  headers. 
! fix, pgsql plugin: sql_recovery_backup_host was not being honoured.
  PG_create_backend() now composes a proper conn_string.
! fix, print plugin: increase successful queries number, QN, only if
  the output file was successfully opened.
! fix, zmq_common.c: moved ZAP socket initialization inside the ZAP
  handler. See: https://github.com/zeromq/libzmq/issues/3313 .
! fix, util.c: length checks in handle_dynname_internal_strings() were
  reviewed. Existings were not working in absence of starting/trailing
  non-variable strings.
! fix, util.c: use lockf() instead of more problematic flock(). Thanks
  to Yuri Lachin ( @yuyutime ) and Miki Takata ( @mikiT ) for their
  support.
! fix, util.c: in compose_timestamp() pad usecs and use "%ld" since
  time fields are signed longs. Thanks to @raymondrussell for the
  patch.
! fix, ndpi_util.c: a protocol bitmask is now set in order to increase
  match rate. Patch is courtesy by @rsolsn.
! fix, compile time warnings: several warnings were addressed including
  but not restricted to -Wreturn-time, -Wunused-variable, implicit func
  declarations, -Wformat-extra-args, -Wunused-label, -Wunused-value,
  -Wunused-function, sbrk calls, -Wpointer-to-int-cast, -Wparentheses
  and -Wint-to-pointer-cast.
! fix, dangerous uninitialized values: net_aggr.c, pmacct.c: in merge()
  argument with non-NULL attribute could be passed NULL; bmp_msg.c: in
  bmp_process_msg_route_monitor() bdata.tstamp could be uninitialized;
  sfprobe_plugin.c: calloc() return value (possibly null) was not being
  checked; sflow_agent.c: uninitialized ret value in sfl_agent_init()
  could lead to undefined bind() error behaviour.
! fix, thread_pool.c: reviewed logics in deallocate_thread_pool() and
  solved a minor memory leak in allocate_thread_pool().
- pmacctd: removed support for FDDI :)
- nfacctd: discontinued support for NetFlow v1, v7 and v8 collection
  and replication.
- pre_tag_map: matching on 'sampling_rate' is not supported anymore as
  a sampling_rate primitive is now available; the 'return' feature to
  return matched data before completing the map workflow has started
  being obsoleted (retired from docs but still available).
- plugin_pipe_check_core_pid: deprecating feature given RabbitMQ and
  Kafka are not supported anymore for internal message delivery.
- tee plugin: obsoleted tee_dissect_send_full_pkt knob, entire packets
  are now replicated only if no pre_tag_map or a simple pre_tag_map is
  defined.
- nfprobe plugin: removed support for NetFlow v1 export.


NOTES.
See UPGRADE file.


Cheers,
Paolo

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to