pmacct is a small set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane data,
ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
and BMP; collect infrastructure data via Streaming Telemetry. Each component
works both as a standalone daemon and as a thread of execution for correlation
purposes (ie. enrich NetFlow with BGP data).
A pluggable architecture allows to store collected forwarding-plane data into
memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
pmacct offers customizable historical data breakdown, data enrichments like
BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are
all supported as inputs for forwarding-plane data. Replication of incoming
NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
easily exported to tools like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP,
Control-plane and infrastructure data, collected via BGP, BMP and Streaming
Telemetry, can be all logged real-time or dumped at regular time intervals
to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
+ BGP, BMP daemons: introduced support for BGP Large Communities IETF
draft (draft-ietf-idr-large-community). Large Communities are stored
in a variable-length field. Thanks to Job Snijders ( @job ) for his
+ BGP daemon: implemented draft-ietf-idr-shutdown. The draft defines a
mechanism to transmit a short freeform UTF-8 message as part of a
Cease NOTIFICATION message to inform the peer why the BGP session is
being shutdown or reset. Thanks to Job Snijders ( @job ) for his
+ tee plugin, pre_tag_map: introduced support for inspetion of specific
flow primitives and selective replication over them. The primitives
supported are: input and output interfaces, source and destination
MAC addresses, VLAN ID. The feature is now limited to sFlow v5 only.
Thanks to Nick Hilliard and Barry O'Donovan for their support.
+ Added src_host_pocode and dst_host_pocode primitives, pocode being a
compact and (de-)aggregatable (easy to identify districts, cities,
metro areas, etc.) geographical representation, based on the Maxmind
v2 City Database. Thanks to Jerred Horsman for his support.
+ Kafka support: introduced support for user-defined (librdkafka) config
file via the new *_kafka_config_file config directives. Full pathname
to a file containing directives to configure librdkafka is expected.
All knobs whose values are string, integer, boolean are supported.
+ AMQP, Kafka plugins: introduced new directives kafka_avro_schema_topic,
amqp_avro_schema_routing_key to transmit Apache Avro schemas at regular
time intervals. The routing key/topic can overlap with the one used to
send actual data.
+ AMQP, Kafka plugins: introduced support for start/stop markers when
encoding is set to Avro (ie. 'kafka_output: avro'); also Avro schema
is now embedded in a JSON envelope when sending it via a topic/routing
key (ie. kafka_avro_schema_topic).
+ print plugin: introduced new config directive avro_schema_output_file
to save the Apache Avro schema in a separate file (it was only possible
to have it combined at the beginning of the data file).
+ BGP daemon: introduced a new bgp_daemon_as config directive to set a
LocalAS which could be different from the remote peer one. This is to
establish an eBGP session instead of a iBGP one (default).
+ flow_to_rd_map: introduced support for mpls_vpn_id. In NetFlow/IPFIX
this is compared against Field Types #234 and #235.
+ sfacctd: introduced support for sFlow v2/v4 counter samples (generic,
ethernet, vlan). This is in addition to existing support for sFlow v5
+ BGP, BMP and Streming Telemetry daemons: added writer_id field when
writing to Kafka and/or RabbitMQ. The field reports the configured
core_proc_name and the actual PID of the writer process (so, while
being able to correlate writes to the same daemon, it's also possible
to distinguish among overlapping writes).
+ amqp, kafka, print plugins: harmonized JSON output to the above: added
event_type field, writer_id field with plugin name and PID.
+ BGP, BMP daemons: added AFI, SAFI information to log and dump outputs;
also show VPN Label if SAFI is MPLS VPN.
+ pmbgpd, pmbmpd: added logics to bypass building RIBs if only logging
BGP/BMP data real-time.
+ BMP daemon: added BMP peer TCP port to log and dump outputs (for NAT
traversal scenarios). Contextually, multiple TCP sessions per IP are
now supported for the same reason.
+ SQL plugins: ported (from print, etc. plugins) the 1.6.1 re-working of
the max_writers feature.
+ uacctd: use current time when we don't have a timestamp from netlink.
We only get a timestamp when there is a timestamp in the skb. Notably,
locally generated packets don't get a timestamp. The patch is courtesy
by Vincent Bernat ( @vincentbernat ).
+ build system: added configure options for partial linking of binaries
with any selection/combination of IPv4/IPv6 accounting daemons, BGP
daemon, BMP daemon and Streaming Telemetry daemon possible. By default
all are compiled in.
+ BMP daemon: internal code changes to pass additional info from BMP
per-peer header to bgp_parse_update_msg(). Goal is to expose further
info, ie. pre- vs post- policy, when logging or dumping BMP info.
! fix, BGP daemon: introduced parsing of IPv6 MPLS VPN (vpnv6) NLRIs.
Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue.
! fix, BGP daemon: upon doing routes lookup, now correctly honouring
the case of BGP-LU (SAFI_MPLS_LABEL).
! fix, BGP daemon: send BGP NOTIFICATION out in case of known failures
! fix, kafka_partition, *_kafka_partition: default value changed from 0
(partition zero) to -1 (RD_KAFKA_PARTITION_UA, partition unassigned).
Thanks to Johan van den Dorpe ( @johanek ) for his support.
! fix, pre_tag_map: removed constraint for 'ip' keyword for nfacctd and
sfacctd maps. While this is equivalent syntax to specifying rules with
'ip=0.0.0.0/0', it allows for map indexing (maps_index: true).
! fix, bgp_agent_map: improved sanity check against bgp_ip for IPv6
addresses (ie. an issue appeared for the case of '::1' where the first
64 bits are zeroed out). Thanks to Charlie Smurthwaite ( @catphish )
for reporting the issue.
! fix, maps_index: indexing now correctly works for IPv6 pre_tag_map
entries. That is, those where 'ip', the IP address of the NetFlow/
IPFIX/sFlow exporter, is an IPv6 address.
! fix, pre_tag_map: if mpls_vpn_rd matching condition is specified and
maps_index is enabled, PT_map_index_fdata_mpls_vpn_rd_handler() now
picks the right (and expected) info.
! fix, pkt_handlers.c: improved definition and condition to free() in
bgp_ext_handler() in order to prevent SEGVs. Thanks to Paul Mabey for
! fix, kafka_common.c: removed waiting time from p_kafka_set_topic().
Added docs advicing to create in advance Kafka topics.
! fix, sfacctd, sfprobe: tag and tag2 are now correctly re-defined as
64 bits long.
! fix, sfprobe plugin, sfacctd: tags and class primitives are now being
encoded/decoded using enterprise #43874, legit, instead of #8800, that
was squatted back in the times. See issue #71 on GiHub for more info.
! fix, sfacctd: lengthCheck() + skipBytes() were producing an incorrect
jump in case of unknown flow samples. Replaced by skipBytesAndCheck().
Thanks to Elisa Jasinska ( @fooelisa ) for her support.
! fix, pretag_handlers.c: in bgp_agent_map added case for 'vlan and ...'
! fix, BGP daemon: multiple issues of partial visibility of the stored
RIBs and SEGVs when bgp_table_per_peer_buckets was not left default:
don't mess with bms->table_per_peer_buckets given the multi-threaded
scenario. Thanks to Dan Berger ( @dfberger ) for his support.
! fix, BGP, BMP daemons: bgp_process_withdraw() function init aligned to
bgp_process_update() in order to prevent SEGVs. Thanks to Yuri Lachin
for his support.
! fix, bgp_msg.c: Route Distinguisher was stored and printed incorrectly
when of type RD_TYPE_IP. Thanks to Alberto Santos ( @m4ccbr ) for
reporting the issue.
! fix, bgp_logdump.c: p_kafka_set_topic() was being wrongly applied to
an amqp_host structure (instead of a kafka_host structure). Thanks to
Corentin Neau ( @weyfonk ) for reporting the issue.
! fix, BGP daemon: improved BGP next-hop setting and comparison in cases
of MP_REACH_NLRI and MPLS VPNs. Many thanks to both Catalin Petrescu
( @cpmarvin ) and Alberto Santos ( @m4ccbr ) for their support.
! fix, pmbgpd, pmbmpd: pidfile was not written even if configured. Thanks
to Aaron Glenn ( @aaglenn ) for reporting the issue.
! fix, tee plugin: tee_max_receiver_pools is now correctly honoured and
debug message shows the replicatd protocol, ie. NetFlow/IPFIX vs sFlow.
! AMQP, Kafka plugins: separate JSON objects, newline separated, are
preferred to JSON arrays when buffering of output is enabled (ie.
kafka_multi_values) and output is set to JSON. This is due to quicker
serialisation performance shown by the Jansson library.
! build system: switched to enable IPv6 support by default (while the
--disable-ipv6 knob can be used to reverse the behaviour). Patch is
courtesy by Elisa Jasinska ( @fooelisa ).
! build system: given visibility, ie. via -V CL option, into compile
options enabled by default (ie. IPv6, threads, 64bit counters, etc.).
! fix, nfprobe: free expired records when exporting to an unavailable
collector in order to prevent a memory leak. Patch is courtersy by
Vladimir Kunschikov ( @kunschikov ).
! fix, AMQP plugin: set content type to binary in case of Apache Avro
! fix, AMQP, Kafka plugins: optimized amqp_avro_schema_routing_key and
kafka_avro_schema_topic. Avro schema is built only once at startup.
! fix, cfg.c: improved parsing of config key-values where squared brakets
appear in the value part. Thanks to Brad Hein ( @regulatre ) for
reporting the issue. Also, detection of duplicates among plugin and
core process names was improved.
! fix, misc: compiler warnings: fix up missing includes and prototypes;
the patch is courtesy by Tim LaBerge ( @tlaberge ).
! kafka_consumer.py, amqp_receiver.py: Kafka, RabbitMQ consumer example
scripts have been greatly expanded to support posting to a REST API or
to a new Kafka topic, including some stats. Also conversion of multiple
newline-separated JSON objects to a JSON array has been added. Misc
bugs were fixed.
See UPGRADE file.
pmacct-discussion mailing list