Re: [pmacct-discussion] pmacct 1.7.4 released !
Hi Olivier, Thanks for reporting this. We could not yet make progress with Fabien as he is busy; but i have his config. Any chance you can send me privately your config so i may start drafting a bottom line among the two? Paolo On Mon, Jan 27, 2020 at 10:42:51PM +0100, Olivier Benghozi wrote: > Hi ! > > oom-killer just killed my instance, so «same here»... > > Some infos: > > > # src/nfacctd -V > NetFlow Accounting Daemon, nfacctd 1.7.4-git (20191126-01+c5) > > Arguments: > '--enable-jansson' '--enable-64bit' '--enable-zmq' '--enable-pgsql' > '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > '--enable-st-bins' > > Libs: > libpcap version 1.8.1 > PostgreSQL 90615 > jansson 2.9 > ZeroMQ 4.2.1 > > System: > Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 > > Compiler: > gcc 6.3.0 > > > > Le 7 janv. 2020 à 09:06, Fabien VINCENT a écrit : > > > > Hi Paolo, > > Thanks for this release and enhancements ! > > Since upgrade, I see a huge memory leak without any reason. > > https://github.com/pmacct/pmacct/issues/356 > > The only changes is I dist-upgrade the machine itself as installed from > > source 1.7.4 release. > > I use print plugin on my side on nfacctd processes. > > Please let me know how I can troubleshoot this, I will rollback to 1.7.3 > > temporarly > > Regards, > > > > Le 31-12-2019 17:48, Paolo Lucente a écrit : > >> VERSION. > >> 1.7.4 > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct 1.7.4 released !
Hi ! oom-killer just killed my instance, so «same here»... Some infos: # src/nfacctd -V NetFlow Accounting Daemon, nfacctd 1.7.4-git (20191126-01+c5) Arguments: '--enable-jansson' '--enable-64bit' '--enable-zmq' '--enable-pgsql' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins' Libs: libpcap version 1.8.1 PostgreSQL 90615 jansson 2.9 ZeroMQ 4.2.1 System: Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 Compiler: gcc 6.3.0 > Le 7 janv. 2020 à 09:06, Fabien VINCENT a écrit : > > Hi Paolo, > Thanks for this release and enhancements ! > Since upgrade, I see a huge memory leak without any reason. > https://github.com/pmacct/pmacct/issues/356 > The only changes is I dist-upgrade the machine itself as installed from > source 1.7.4 release. > I use print plugin on my side on nfacctd processes. > Please let me know how I can troubleshoot this, I will rollback to 1.7.3 > temporarly > Regards, > > Le 31-12-2019 17:48, Paolo Lucente a écrit : >> VERSION. >> 1.7.4 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct 1.7.4 released !
Hi Paolo, Thanks for this release and enhancements ! Since upgrade, I see a huge memory leak without any reason. https://github.com/pmacct/pmacct/issues/356 The only changes is I dist-upgrade the machine itself as installed from source 1.7.4 release. I use print plugin on my side on nfacctd processes. Please let me know how I can troubleshoot this, I will rollback to 1.7.3 temporarly Regards, Le 31-12-2019 17:48, Paolo Lucente a écrit : VERSION. 1.7.4 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI data; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.4.tar.gz CHANGELOG. + Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder for multi-vendor environments written in Python3. It supports gRPC transport along with Protobuf encoding as input and can output to Kafka with Avro encoding. Output to files and JSON encoding is currently supported sending data via ZMQ to pmtelemetryd first. It was tested working with data input from Cisco and Huawei routers and v3 replaces v2. Thanks to the Streaming Telemetry core team: Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ), Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ). + Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.) have been wired to the VXLAN decoding and new ones (tunnel_src_mac, tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined. + BMP daemon: added support for Peer Up message namespace for TLVs (draft-ietf-grow-bmp-peer-up) and also support for Route Monitor and Peer Down TLVs (draft-ietf-grow-bmp-tlv). + BGP, BMP daemons: in addition to existing JSON export, data can now be exported in Apache Avro format. There is also support for the Confluent Schema Registry. + Introduced support for JSON-encoded Apache Avro encoding. While the binary-encoded Apache Avro is always recommended for any production scenarios (also to optionallly leverage Confluent Schema Registry support), JSON-encoded is powerful for testing and troubleshooting scenarios. + sfprobe plugin: added support for IPv6 transport for sFlow export. sfprobe_agentip is an IP address put in the header of the sFlow packet. If underlying transport is IPv6, this must be configured to an IPv6 address. + zmq_common.[ch]: Improved modularity of the ZMQ internal API and decoupled bind/connect from push/pull and pub/sub; also improved support for inproc sockets. All to increase the amount of use-cases covered by the API. + bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use cases. + nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to override ifindexes dynamically determined (ie. by NFLOG) with values computed by [sn]fprobe_ifindex. + MySQL, PostgreSQL plugins: added support for SSL/TLS connections by specifying a CA certificate (sql_conn_ca_file). + Kafka, AMQP plugins: amqp_markers and kafka_markers have now been properly re-implemented when output encoding is Avro using an own Avro schema (instead of squatting pieces of JSON in the data stream for the very purpose). + print plugin: introduced print_write_empty_file config knob (true, false) to create an empty output file when there are no cache entries to purge. Such behaviour was present in versions up to 0.14 and may be preferred by some to the new >= 1.5 versions behaviour. Thanks to Lee Yongjae ( @setup74 ) for the con
[pmacct-discussion] pmacct 1.7.4 released !
VERSION. 1.7.4 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI data; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). A pluggable architecture allows to store collected forwarding-plane data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. pmacct offers customizable historical data breakdown, data enrichments like BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are all supported as inputs for forwarding-plane data. Replication of incoming NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be easily exported to time-series databases like ElasticSearch and InfluxDB and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. Control-plane and infrastructure data, collected via BGP, BMP and Streaming Telemetry, can be all logged real-time or dumped at regular time intervals to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-1.7.4.tar.gz CHANGELOG. + Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder for multi-vendor environments written in Python3. It supports gRPC transport along with Protobuf encoding as input and can output to Kafka with Avro encoding. Output to files and JSON encoding is currently supported sending data via ZMQ to pmtelemetryd first. It was tested working with data input from Cisco and Huawei routers and v3 replaces v2. Thanks to the Streaming Telemetry core team: Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ), Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ). + Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.) have been wired to the VXLAN decoding and new ones (tunnel_src_mac, tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined. + BMP daemon: added support for Peer Up message namespace for TLVs (draft-ietf-grow-bmp-peer-up) and also support for Route Monitor and Peer Down TLVs (draft-ietf-grow-bmp-tlv). + BGP, BMP daemons: in addition to existing JSON export, data can now be exported in Apache Avro format. There is also support for the Confluent Schema Registry. + Introduced support for JSON-encoded Apache Avro encoding. While the binary-encoded Apache Avro is always recommended for any production scenarios (also to optionallly leverage Confluent Schema Registry support), JSON-encoded is powerful for testing and troubleshooting scenarios. + sfprobe plugin: added support for IPv6 transport for sFlow export. sfprobe_agentip is an IP address put in the header of the sFlow packet. If underlying transport is IPv6, this must be configured to an IPv6 address. + zmq_common.[ch]: Improved modularity of the ZMQ internal API and decoupled bind/connect from push/pull and pub/sub; also improved support for inproc sockets. All to increase the amount of use-cases covered by the API. + bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use cases. + nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to override ifindexes dynamically determined (ie. by NFLOG) with values computed by [sn]fprobe_ifindex. + MySQL, PostgreSQL plugins: added support for SSL/TLS connections by specifying a CA certificate (sql_conn_ca_file). + Kafka, AMQP plugins: amqp_markers and kafka_markers have now been properly re-implemented when output encoding is Avro using an own Avro schema (instead of squatting pieces of JSON in the data stream for the very purpose). + print plugin: introduced print_write_empty_file config knob (true, false) to create an empty output file when there are no cache entries to purge. Such behaviour was present in versions up to 0.14 and may be preferred by some to the new >= 1.5 versions behaviour. Thanks to Lee Yongjae ( @setup74 ) for the contribution. ! fix, signals.c: signals handling has been restructured in order to block certain signals during critical sections of data processing. Thanks to Vaibhav Phatarpekar ( @vphatarp ) for the contribution. ! fix, signals.c: slimmed reload() signal handler code and moved it to a synchronous section. The handler is to reset logging output to files or syslog. Thanks to Jared Mauch ( @jaredmauch ) for his support resolving this. ! fix, pmbgpd, pmbmpd