pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry.
A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP,
Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
+ Introduced pmbgpd daemon: a stand-alone BGP collector daemon; acts as a
passive neighbor and maintains per-peer RIBs; can log real-time and/or
dump at regular time-intervals BGP data to configured backends.
+ Introduced pmbmpd daemon: a stand-alone BMP collector daemon; can log
real-time and/or dump at regular time-intervals BMP and BGP data to
+ Introduced Apache Avro as part of print, AMQP and Kafka output: Apache
Avro is a data serialization system providing rich data structures, a
compact, fast, binary data format, a container file to store persistent
data, remote procedure call (RPC) and simple integration with dynamic
languages. The implementation is courtesy by Codethink Ltd.
+ as_path, std_comm and ext_comm primitives: along with their src counter
parts, ie. src_as_path etc., have been re-worked to a variagle-length
internal representation which will lead, when using BGP primitives, to
memory savings of up to 50% compared to previous releases.
+ std_comm, ext_comm primitives: primitives are de-coupled so that they
are not multiplexed anymore in the same field, on output. Added a
tmp_comms_same_field config directive for backward compatibility.
+ nfacctd: added support for repeated NetFlow v9/IPFIX field types. Also
flowStartDeltaMicroseconds (IE #158) and flowEndDeltaMicroseconds (#159)
are now supported for timestamping.
+ kafka plugin: it is now possible to specify -1 (RD_KAFKA_RTITION_UA) as
part of the kafka_partition config directive. Also, introduced support
for Kafka partition keys via kafka_partition_key and equivalent config
+ kafka plugin: kafka_broker_host directive now allows to specify multiple
brokers, ie. "broker1:10000,broker2". The feature relies on capabilities
of underlying rd_kafka_brokers_add().
+ tee, nfprobe, sfprobe plugins: introduced Kafka support for internal
pipe and buffering, ie. plugin_pipe_kafka. This is in addition to the
existing support for homegrown internal buffering and RabbitMQ.
+ tee plugin: introduced support for variable-length buffers which reduces
+ print, MongoDB, AMQP and Kafka plugins: re-worked max_writers feature to
not rely anymore on waitpid() inside signal handlers as it was failing on
some OS versions (and could not be reproduced on others). Thanks to
Janet Sullivan for her support.
+ bgp_follow_nexthop_external: introduced feature to return, when true, the
next-hop from the routing table of the last node part of the supplied IP
prefix(es) as value for the 'peer_ip_dst' primitive. When false, default,
it returns the IP address of the last node part of the bgp_follow_nexthop
+ pmtelemetryd: added initial support for GPB. Input GPB data is currently
base64'd in the telemetry_data field of the daemon output JSON object.
+ pmtelemetryd: Added telemetry statistics. For each peer, track the number
of packets received, how many bytes are pulled off the wire, and the
resulting message payload. Dump these counts in logdump. Patch is courtesy
by Tim LaBerge.
+ amqp_markers, kafka_markers: added start/end markers feature to AMQP and
Kafka plugins output same as for the print plugin (print_markers).
+ pre_tag_map: 'direction' keyword now applies to sFlow too: it does expect
values 0 (ingress direction) or 1 (egress direction), just like before.
In sFlow v2/v4/v5 this returns a positive match if: 1) source_id equals
to input interface and this 'direction' key is set to '0' or 2) source_id
equals to output interface and this 'direction' key is set to '1'.
+ bgp_agent_map: introduced support for input and output interfaces. This
is relevant to VPN scenarios.
+ tmp_asa_bi_flow hack: bi-flows use two counters to report counters, ie.
bytes and packets, in forward and reverse directions. This hack (ab)uses
the packets field in order to store the extra bytes counter.
! fix, nfacctd: debugging NetFlow v9/IPFIX templates, added original field
type number to the output when the field is known and its description is
! fix, Jansson: added JSON_PRESERVE_ORDER flag to json_dumps() to give
output consistency across runs.
! fix, kafka_common.c: added rd_kafka_message_destroy() to p_kafka_consume_
_data() to prevent memory leaks. Thanks to Paul Mabey for his support
solving the issue.
! fix, kafka_common.c: p_kafka_set_topic() now gives it some time for the
topic to get (auto) created, if needed.
! fix, print plugin: improved check for when to print table title (csv,
formatted). Either 1) print_output_file_append is set to false or 2)
print_output_file_append is set to true and file is to be created.
! fix, print_markers: start marker is now printed also in the case where
print_output_file_append is set to true. Also, markers are now printed as
a JSON object, if output is set to JSON.
! fix, pkt_handlers.c: removed l3_proto checks from NF_peer_dst_ip_handler()
for cases where a v6 flows has a v4 BGP next-hop (ie. vpnv6)
! fix, pre_tag_map: removed 32 chars length limit from set_label statement.
! fix, custom primitives: names are now interpreted as case-insensitive.
Patch is courtesy by Corentin Neau.
! fix, BGP, BMP and Streaming Telemetry: if reopening [bgp, bmp, telemetry]_
daemon_msglog_file via SIGHUP, reset reload flag.
! fix, BGP, BMP and Streaming Telemetry: removed gettimeofday() from bgp_
peer_dump_init() and bgp_peer_dump_close() in order to maintain a single
timestamp for a full dump event. Thanks to Tim LaBerge for his support.
! fix, BGP, BMP and Streaming Telemetry: output log and dump messages went
through a general review to improve information consistency and usability.
Message formats are now documented in docs/MSGLOG_DUMP_FORMATS so to more
easily track future changes.
! fix, pmtelemetryd: avoiding un-necessary spawn of a default plugin if none
! fix, pmtelemetryd: Mask SIGCHLD during socket IO. If we happen to be
blocked in recv() while a log dump happens, recv() will fail with EINTR.
This is to mask SIGCHLD during socket IO and restores the original mask
after the IO completes. Patch is courtesy by Tim LaBerge.
! fix, build system: misc improvements made to the build system introduced
in 1.6.0. Thanks to Vincent Bernat for his support in this area.
! fix, compiler warnings: ongoing effort to suppress warning messages when
compiling. Thanks to Tim LaBerge, Matin Mitchell for their contributions.
See UPGRADE file.
pmacct-discussion mailing list