Hi Paolo,
Thanks for this release and enhancements !
Since upgrade, I see a huge memory leak without any reason.
The only changes is I dist-upgrade the machine itself as installed from source 1.7.4 release.
I use print plugin on my side on nfacctd processes.
Please let me know how I can troubleshoot this, I will rollback to 1.7.3 temporarly

Le 31-12-2019 17:48, Paolo Lucente a écrit :

pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI data; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with
BGP data).

A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to time-series databases like ElasticSearch and InfluxDB and
traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.

Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.



+ Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder
  for multi-vendor environments written in Python3. It supports gRPC
  transport along with Protobuf encoding as input and can output to
  Kafka with Avro encoding. Output to files and JSON encoding is
  currently supported sending data via ZMQ to pmtelemetryd first. It
  was tested working with data input from Cisco and Huawei routers
  and v3 replaces v2. Thanks to the Streaming Telemetry core team:
  Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ),
  Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ).
+ Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic
daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel
  primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.)
  have been wired to the VXLAN decoding and new ones (tunnel_src_mac,
  tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined.
+ BMP daemon: added support for Peer Up message namespace for TLVs
  (draft-ietf-grow-bmp-peer-up) and also support for Route Monitor
  and Peer Down TLVs (draft-ietf-grow-bmp-tlv).
+ BGP, BMP daemons: in addition to existing JSON export, data can now
  be exported in Apache Avro format. There is also support for the
  Confluent Schema Registry.
+ Introduced support for JSON-encoded Apache Avro encoding. While the
  binary-encoded Apache Avro is always recommended for any production
  scenarios (also to optionallly leverage Confluent Schema Registry
  support), JSON-encoded is powerful for testing and troubleshooting
+ sfprobe plugin: added support for IPv6 transport for sFlow export.
  sfprobe_agentip is an IP address put in the header of the sFlow
  packet. If underlying transport is IPv6, this must be configured to
  an IPv6 address.
+ zmq_common.[ch]: Improved modularity of the ZMQ internal API and
  decoupled bind/connect from push/pull and pub/sub; also improved
  support for inproc sockets. All to increase the amount of use-cases
  covered by the API.
+ bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use
+ nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to
  override ifindexes dynamically determined (ie. by NFLOG) with values
  computed by [sn]fprobe_ifindex.
+ MySQL, PostgreSQL plugins: added support for SSL/TLS connections by
  specifying a CA certificate (sql_conn_ca_file).
+ Kafka, AMQP plugins: amqp_markers and kafka_markers have now been
  properly re-implemented when output encoding is Avro using an own
  Avro schema (instead of squatting pieces of JSON in the data stream
  for the very purpose).
+ print plugin: introduced print_write_empty_file config knob (true,
  false) to create an empty output file when there are no cache entries
  to purge. Such behaviour was present in versions up to 0.14 and may
  be preferred by some to the new >= 1.5 versions behaviour. Thanks to
  Lee Yongjae ( @setup74 ) for the contribution.
! fix, signals.c: signals handling has been restructured in order to
  block certain signals during critical sections of data processing.
  Thanks to Vaibhav Phatarpekar ( @vphatarp ) for the contribution.
! fix, signals.c: slimmed reload() signal handler code and moved it to
  a synchronous section. The handler is to reset logging output to
  files or syslog. Thanks to Jared Mauch ( @jaredmauch ) for his
  support resolving this.
! fix, pmbgpd, pmbmpd and pmtelemetryd daemons: added extra signals
  handling (SIGINT, SIGTERM, SIGCHLD) consistently to traffic daemons.
! fix, BGP daemon: withdrawals of label-unicast (support introduced in
  1.7.3) and mpls-vpn NLRIs did fail to parse in release 1.7.3 and
  were silently discarded.
! fix, nfacctd: wired (BGP, BMP, ISIS, etc) lookups to NetFlow (Secure)
  Event Logging (NEL/NSEL).
! fix, pmtelemetryd: re-implemented a decoder for so-called Cisco v1
  Streaming Telemetry proprietary header over UDP/TCP streams.
! fix, pmtelemetryd: improved sanitization of input JSON objects by
  also checking for isspace() other than isprint() for pretty-printed
! maps_index: optimized lookups, improved debugging output upon loading
! fix, tee plugin: overwriting computed IP address length with socket
  container length was found to prevent output data on some BSDs.
! fix, kafka_common.c: if taking the p_kafka_close() route, ensure to
  return and not perform any further polling in order to avoid SEGVs.
! fix, BMP daemon: incorrect decoding of type was preventing correct
  logging of Init and Term messages extra info. Also in Term messages
  TLV data was incorrectly consumed twice triggering length check
! fix, BMP daemon: added checks for successful BGP PDU parsing in both
  Peer Up (BGP OPEN) and Route Monitor (BGP UPDATE) messages.
! fix, BMP daemon: improved length checks and making sure that strings
  potentially non null-terminated are now terminated. Also TLV-related
  code has been refactored.
! fix, pmbgp.py: the example client for BGP Looking Glass was migrated
  to Python3: thanks to @brusilov for the contribution.
! fix, nfacctd: if src_port or dst_port primitives are selected, enable
  IP fragment handling. Needed to process L4 of IPFIX IE #351.
! fix, nfv9_template.c: correct handling of variable-length IPFIX
  fields. Thanks to Nimrod Mesika ( @nimrody ) for the contribution.
! fix, PostgreSQL plugin: ABSTIME was replaced with to_timestamp() in
  queries as support for ABSTIME was dropped as of PostgreSQL 12. Many
  thanks to Manuel Mendez ( @mmlb ) for the contribution.
! fix, PostgreSQL plugin: SEGVs were observed when the queue of pending
  queries was non-empty (ie. nfacctd_time_new set to false, default);
  thanks to Guo-Wei Su ( @nansenat16 ) for the contribution.
! fix, cfg_handlers: [sn]facctd_disable_checks, nfacctd_disable_opt_
  scope_check could not be properly set to false.
! fix, sql_common.c: src_host_coords and dst_host_coords primitives
  have been correctly spaced in SQL queries. Also float values are now
  quoted. Finally, sampling_direction primitive is encoded correctly.
! fix, kafka plugin: if kafka_avro_schema_registry is in use, subject
  name is aligned to Kafka topic name (if topic is not dynamic).
! fix, pretag.c: when using 'label', store the label string in the
  heap (instead of the stack). Thanks to Raphael P. Barazzutti
  ( @rbarazzutti ) for the contribution.
! fix, pretag.c: JEQ labels are now correctly free() during init upon
  map reload.
! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was
  causing plugin_pipe_zmq operations to fail on certain compilers (ie.
  gcc7). Thanks to Yuri Lachin ( @yuyutime ) for his support.
! fix, cfg_handlers.c: reviewed handling of parsed 'zero' value for
  several config directives.
! fix, countless code warnings when enabling -Wall (--enable-debug);
  also included -Wall in Continuous Integration tests. Restructured
  globals, header inclusions, function prototypes definition, etc.
  Many thanks to Marc Sune ( @msune ) for all his efforts.
! fix, configure.ac: evaluation of --enable-debug pushed to the end of
  the script so to not interfere with tests (ie. alignment, endianess,
- BMP daemon: retired support for draft-hsmit-bmp-extensible-routemon-
- AMQP plugin: obsoleted amqp_avro_schema feature (which includes
  amqp_avro_schema_routing_key and amqp_avro_schema_refresh_time keys
  config keys). Avro schemas can now only be written to files.

1.7.3 -- 16-05-2019
+ Introduced the RPKI daemon to build a ROA database and check prefixes
  validation status and coverages. Resource Public Key Infrastructure
  (RPKI) is a specialized public key infrastructure (PKI) framework
  designed to secure the Internet routing. RPKI uses certificates to
  allow Local Internet Registries (LIRs) to list the Internet number
  resources they hold. These attestations are called Route Origination
  Authorizations (ROAs). ROA information can be acquired in one of the
  two following ways: 1) importing it using the rpki_roas_file config
  directive from a file in the RIPE Validator format or 2) connecting
  to a RPKI RTR Cache for live ROA updates; the cache IP address/port
  being defined by the rpki_rtr_cache config directive (and a few more
  optional rpki_rtr_* directives are available and can be reviwed in
  the CONFIG-KEYS doc). The ROA fields will be populated with one of
  these five values: 'u' Unknown, 'v' Valid, 'i' Invalid no overlaps,
  'V' Invalid with a covering Valid prefix, 'U' Invalid with a covering
  Unknown prefix. Thanks to Job Snijders ( @job ) for his support and
+ Introducing pmgrpcd.py, written in Python, a daemon to handle gRPC-
  based Streaming Telemetry sessions and unmarshall GPB data. Code
  was mostly courtesy by Matthias Arnold ( @tbearma1 ). This is in
  addition (or feeding into) pmtelemetryd, written in C, a daemon to
  handle TCP/UDP-based Streaming Telemetry sessions with JSON-encoded
  data. Thanks to Matthias Arnold ( @tbearma1 ) and Thomas Graf for
  their support and contributing code.
+ pmacctd, uacctd: added support for CFP (Cisco FabricPath) and Cisco
  Virtual Network Tag protocols. Both patches were courtesy by Stephen
  Clark ( @sclark46 ).
+ print plugin: added 'custom' to print_output. This is to cover two
  main use-cases: 1) use JSON or Avro encodings but fix the format of
  the messages in a custom way and 2) use a different encoding than
  JSON or Avro. See also example in examples/custom and new directives
  print_output_custom_lib and print_output_custom_cfg_file. The patch
  was courtesy by Edge Intelligence ( @edge-intelligence ).
+ Introducing mpls_pw_id aggregation primitive and mpls_pw_id key in
  pre_tag_map to filter on signalled L2 MPLS VPN Pseudowire IDs.
+ BGP daemon: added bgp_disable_router_id knob to enable/disable BGP
  Router-ID check, both at BGP OPEN time and BGP lookup. Useful, for
  example, in scenarios with split BGP v4/v6 AFs over v4/v6 transports.
+ BGP, BMP daemons: translate origin attribute numeric value into IGP
  (i), EGP (e) and Incomplete (u) strings.
+ plugins: added new plugin_exit_any feature to make the daemon bail
  out if any (not all, which is the default behaviour) of the plugins
+ maps_index: improved selection of buckets for index hash structure
  by picking the closest prime number to the double of the entries of
  the map to be indexed in order to achieve better elements dispersion
  and hence better performances.
+ nfacctd: added support for IPFIX templateId-scoped (IE 145) sampling
+ pmacctd, uacctd, sfacctd, nfacctd: added a -M command-line option to
  set *_markers (ie. print_markers) to true and fixed -A command-line
  option to set print_output_file_append to align to true/false.
! fix, BGP, BMP, Streaming Telemetry daemons: improved sequencing of
  dump events by assigning a single sequence number per event (ie. for
  streaming pipeline scenarios in order to reduce correlation with
  dump_init/dump_close messages). Also amount of record dumped was
  added to the close message.
! fix, BGP, BMP, Streaming Telemetry daemons: removed hierarchical
  json_decref() since json_object_get() borrows reference. This was
  occasionaly leading to SEGVs.
! fix, uacctd: dynamically allocate jumbo_container buffer size as
  packets larger than 10KB, previous static allocation, would lead to
! fix, nfacctd: wired (BGP, BMP, ISIS, etc.) lookups to the NEL/NSEL
! fix, nfacctd: search for IE 408 (dataLinkFrameType) was leading to
  SEGVs. Also improved handling of variable-length IPFIX templates.
! fix, BMP daemon: solved an occasional truncation of the last message
  in a packet.
! fix, BGP daemon: when processing bgp_daemon_md5_file, ipv4 addresses
  were incorrectly translated to ipv4-mapped ipv6 ones as a result of
  which TCP-MD5 hashes were not correctly bound to sockets.
! fix, BGP daemon: improved label-unicast and mpls-vpn SAFIs handling
  (some bogus messages, multiple labels, etc.).
! fix, BGP daemon: introduced PREFIX_STRLEN to make enough room for
  prefix2str() calls (before unsufficient INET6_ADDRSTRLEN was used).
! fix, BMP daemon: improved handling of ADD-PATH capability.
! fix, plugins: an incorrect evaluation in P_cache_attach_new_node did
  make possible to buffer overrun in plugins cache allocation. This was
  found related to a "[..]: Assertion `!cache_ptr->stitch' failed."
  daemon bail-out message.
! fix, plugins: if pidfile directive was enabled, exit_gracefully() was
  mistakenly deleting the plugin pidfile when called by a child process
  (ie. writer, dumper, etc.).
! fix, plugins: when taking exit_gracefully(), if the process is marked
  as 'is_forked', just exit and don't perform extra ops in exit_all()
  or exit_plugin().
! fix, plugins: re-evaluate dynamic tables/files name if *_refresh_time
  is different than *_history period.
! fix, SQL plugins: a missing 'AND' was making SQL statements related
  to src_host_coords and dst_host_coords fail.
! fix, GeoIPv2: if no match is returned by libmaxminddb, return O1 code
  (Other Country) instead of a null value.
! fix, flow_to_rd_map: mpls_vpn_id was not working when maps_index was
  enabled. Also partly re-written mpls_vpn_id handler.
! fix, nfprobe plugin: serialize_bin() function introduced for correct
  serialization of custom primitives defined with 'raw' semantics.
! fix, PostgreSQL plugin: testing for presence of PQlibVersion() in
  libpq to prevent compiling issues (ie. on CentOS 6).
! fix, MySQL plugin: including mysql_version.h to compile successfully
  against newer MariaDB releases.
! fix, nDPI classification: send log message if 'class' primitive is
  selected but nDPI is not compiled in; also updated code to follow
  API changes in versions >= 2.6 of the library. Dropped support for
  versions < 2.4.
! fix, sfprobe plugin: added (and documented) conditional for optional
  export of classification info.
! fix, aggregate_primitives: field_type is now also allowed for pmacctd
  and uaccd daemons so that it can be used for NetFlow v9/IPFIX export
  (nfprobe plugin) purposes.
! fix, pre_tag_map: if no 'ip' keyword is specified, an entry of the
  map gets recirculated in order to be set for both v4 and v6 maps. If
  a 'set_label' is also specified, it was causing a SEGV. Now the label
  is correctly copied in case of recirculation.
! fix, zmq_common.c: added option for non-blocking p_zmq_send_bin() as
  otherwise program would block in case of no consumers (main use-case:
  flow replication over ZeroMQ queues); as a result, a generous hwm
  value was added on both sides of these queues.
! fix, zmq_common.c: ZAP socket moved inside thread to prevent failed
  assert() when compiling with gcc7/gcc8. Also a single user/password
  auto-generated combination is used for all plugins.
! fix, signals.c: SIGUSR1 handler for nfacctd and nfacctd is changed to
  syncronous in order to prevent race conditions. Also, in pmacctd,
  upon sending SIGUSR1, stats were not printed when reading packets
  from a pcap_interfaaces_map.
! fix, plugin_cmn_json.c: if leaving protocols numerical (ie. proto,
  tunnel_proto primitives), convert them to string-represented numbers
  for data consistency for consumers.
! fix, util.c: open_output_file(), if file exists and it's a FIFO then
  set O_NONBLOCK when opening.
! fix, pretag.c: pretag_index_report() was reporting incorrect info of
  the hash structure built for the maps_index feature. Its format was
  has also changed to be better parseable.
! fix, compile time warnings: several warnings were addressed including
  but not restricted to -Wformat ones. Also an annotation was added to
  the Log function to inform the compiler it's a printf-style function,
  allowing it to give warnings for argument mismatches.
- --enable-ipv6 configure script switch has been deprecated and, as a
  result, IPv6 support was made mandatory.
- BGP daemon: removed unused pathlimit field from bgp_attr structure.
- pmacct client: removed deprecated SYM field from from formatted and
  CSV headers.

See UPGRADE file.


pmacct-discussion mailing list

pmacct-discussion mailing list

Reply via email to