Hi Ed, You mean you kind of just upgraded to 1.5.2 and you were not seeing the same before the upgrade? If you leave it running, does it grow undefinitely or at some stage it looks it reaches a stable memory footprint? Especially if using malloc()ed structures, ie. derived by BGP primitives, this could be a valid scenario. Finally, can you share your config?
Cheers, Paolo On Thu, Jan 07, 2016 at 04:04:09PM -0600, Edward Henigin wrote: > Hi Paolo, > > I'm seeing a significant memory leak in the print plugin in 1.5.2. > > # service nfacctd restart > Shutting down sflow collection services (nfacctd): [ OK ] > Starting sflow collection services (nfacctd): INFO ( default/core ): > Daemonizing. Hmm, bye bye screen. > [ OK ] > # pgrep nfacctd > 874 > 876 > # while [ 1 ]; do sleep 60; date; ps uwp 876; done > Thu Jan 7 15:53:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.1 0.6 115276 104432 ? S 15:52 0:01 nfacctd: > Print Plugin [default] > Thu Jan 7 15:54:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.0 0.6 120884 109980 ? S 15:52 0:01 nfacctd: > Print Plugin [default] > Thu Jan 7 15:55:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.0 0.7 128408 117608 ? S 15:52 0:02 nfacctd: > Print Plugin [default] > Thu Jan 7 15:56:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.1 0.8 147680 136856 ? S 15:52 0:03 nfacctd: > Print Plugin [default] > Thu Jan 7 15:57:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.1 0.9 160748 149944 ? S 15:52 0:03 nfacctd: > Print Plugin [default] > Thu Jan 7 15:58:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.1 1.0 180548 169668 ? S 15:52 0:04 nfacctd: > Print Plugin [default] > Thu Jan 7 15:59:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.1 1.1 203384 192604 ? S 15:52 0:05 nfacctd: > Print Plugin [default] > Thu Jan 7 16:00:56 CST 2016 > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 876 1.3 1.5 272024 261120 ? S 15:52 0:07 nfacctd: > Print Plugin [default] > ^C > # strings - /usr/sbin/nfacctd | fgrep 1.5 > NetFlow Accounting Daemon, nfacctd 1.5.2 > /opt/pmacct/pmacct-1.5.2/src > /opt/pmacct/pmacct-1.5.2/src/nfprobe_plugin > /opt/pmacct/pmacct-1.5.2/src/sfprobe_plugin > /opt/pmacct/pmacct-1.5.2/src/bgp > /opt/pmacct/pmacct-1.5.2/src/tee_plugin > /opt/pmacct/pmacct-1.5.2/src/isis > /opt/pmacct/pmacct-1.5.2/src/bmp > > Any thoughts? Suggestions? > > Thanks, > > Ed > > On Mon, Sep 7, 2015 at 1:01 PM, Paolo Lucente <[email protected]> wrote: > > > VERSION. > > 1.5.2 > > > > > > DESCRIPTION. > > pmacct is a small set of passive network monitoring tools to account, > > classify, > > aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable > > architecture > > allows to store collected data into memory tables, RDBMS (MySQL, > > PostgreSQL, > > SQLite), noSQL databases (MongoDB, BerkeleyDB) AMQP message exchanges > > (RabbitMQ) and flat-files and export through NetFlow or sFlow protocols to > > remote collectors. pmacct offers customizable historical data breakdown, > > BGP > > and IGP correlation, BMP stats and events, GeoIP lookups, sampling and > > renormalization, filtering, tagging and triggers. Libpcap, Linux > > Netlink/ULOG, > > sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are supported. Replication of > > incoming NetFlow and sFlow datagrams is also available. Statistics can be > > easily > > exported to tools like RRDtool, Net-SNMP, MRTG, GNUPlot and Cacti. > > > > > > HOMEPAGE. > > http://www.pmacct.net/ > > > > > > DOWNLOAD. > > http://www.pmacct.net/pmacct-1.5.2.tar.gz > > > > > > CHANGELOG. > > + Introduced support for a RabbitMQ broker to be used for queueing and > > data exchange between Core Process and plugins. This is in alternative to > > the home-grown circular queue implementation. plugin_pipe_amqp directive, > > along with all other plugin_pipe_amqp_* directives, can be set globally > > or apply on a per plugin basis (ie. it is a valid scenario, if multiple > > plugins are instantiated, that some make use of home-grown queueing, > > while others use RabbitMQ based queueing). > > + Introducting support for Maximind GeoIP v2 (libmaxminddb) library: if > > pmacct is compiled with --enable-geoipv2, this defines full pathname to > > a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database > > format is supported (ie. it is not possible to load distinct CSVs for > > IPv4 and IPv6 addresses). > > + Introduced infrastructure for sFlow counters and support specifically for > > generic, ethernet and vlan counters. Counters are exported in JSON format > > to files, specified via sfacctd_counter_file. The supplied filename can > > contain as variable the sFlow agent IP address. > > + Introduced a new thread_stack config directive to allow to modify the > > thread stack size. Natanael Copa reported that some libc implementations, > > ie. musl libc, may set a stack size that is too small by default. > > + Introduced networks_file_no_lpm feature: it applies when aggregation > > method includes src_net and/or dst_net and nfacctd_net (or equivalents) > > and/or nfacctd_as_new (or equivalents) are set to longest (or fallback): > > an IP prefix defined as part of the supplied networks_file wins always, > > even if it's not longest. > > + tee plugin: added support for (non-)transparent IPv6 replication [further > > QA required] > > + plugin_common.c, sql_common.c: added log message to estimate base cache > > memory usage. > > + print, AMQP, MongoDB plugins; sfacctd, BGP, BMP daemons: introducing > > timestamps_since_epoch to write timestamps in 'since Epoch' format. > > + nfacctd: flow bytes counter can now be sourced via element ID #352 > > (layer2OctetDeltaCount) in addition to element ID's already supported. > > Thanks to Jonathan Thorpe for his support. > > + Introducing proc_priority: redefines the process scheduling priority, > > equivalent to using the 'nice' tool. Each daemon process, ie. core, > > plugins, etc., can define a different priority. > > ! fix, BMP daemon: improved preliminar checks in bmp_log_msg() and added > > missing SIGHUP signal handling to reload bmp_daemon_msglog_file files. > > ! fix, bgp_logdump.c: under certain configuration conditions call to both > > write_and_free_json() and write_and_free_json_amqp() was leading to SEGV. > > Thanks to Yuriy Lachin for reporting the issue. > > ! fix, BGP daemon: improved BGP dump output: more accurate timestamping of > > dump_init, dump_close events. dump_close now mentions amount of entries > > and tables dumped. Thanks to Yuriy Lachin for brainstorming around this. > > ! fix, cfg.c: raised amount of allowed config lines from 256 to 8K. > > ! fix, print/AMQP/MongoDB plugins: SEGV observed when IPFIX vlen variables > > were stored in the pending_queries_queue structure (ie. as a result of a > > time mismatch among the IPFIX exporter and the collector box). > > ! fix, vlen primitives: when 'raw' semantics was selected, print_hex() was > > returning wrong hex string length (one char short). As a consequence > > occasionally some extra dirty chars were seen at the end of the converted > > string. > > ! fix, vlen primitives: memory leak verified in print/AMQP/MongoDB plugins. > > ! fix, print, MongoDB & AMQP plugins: dirty values printed as part of the > > 'proto' under certain conditions. Thanks to Rene Stoutjesdijk for his > > support resolving the issue. > > ! fix, amqp_common.c: amqp_exchange_declare() call changed so to address > > the change of rabbitmq-c API for support of auto_delete & internal for > > exchange.declare. Backward compatibility with rabbitmq-c <= 0.5.2 is > > also taken care of. Thanks to Brent Van Dussen for reporting the issue. > > ! fix, compiling on recent FreeBSD: solved some errors catched by the -Wall > > compiler flag. Thanks to Stephen Fulton for reporting the issue. Most of > > the patch is courtesy by Mike Bowie. > > ! fix, print/AMQP/MongoDB plugins: enforcing cleanup of malloc()ed structs > > part of entries added to the pending queue, ie. because seen as future > > entries due to a mismatch of the collector clock with the one of NetFlow/ > > IPFIX exporter(s). This may have lead to data inconsistencies. > > ! fix, amqp_common.c: Content type was only specified for messages > > published > > when the amqp_persistent_msg configuration option is specified. This info > > should always be applied to describe the payload of the message. Patch is > > courtesy by Will Dowling. > > ! fix, amqp_plugin.c: generate an error on compile if --enable-rabbitmq is > > specified without --enable-jansson. It's clear in the documentation that > > both are required for AMQP support, but if built without jansson it will > > silently not publish messages to AMQP. Patch is courtesy by Will Dowling. > > ! fix, amqp_common.c: modified the content type to "application/json" in > > line with RFC4627. Patch is courtesy by Will Dowling. > > ! fix, setsockopt(): u_int64_t pipe_size vars changed to int, in line with > > typical OS buffer limits (Linux, Solaris). Introduced check supplied pipe > > size values are not bigger than INT_MAX. Many thanks to Markus Weber for > > reporting the issue. > > ! fix, nl.c: removed pretag_free_label() from pcap_cb() and ensuring init > > of pptrs. Under certain conditions SEGVs could be noticed. > > ! fix, flow stitching: when print/AMQP/MongoDB plugins were making use of > > the pending queries queue, ie. to compensate for time offsets/flows in > > the future, the stitching feature could potentially lead to SEGV due to > > unsettled pointers. > > ! fix, pgsql plugin: SEGV were noticed when insert/update queries to the > > PostgreSQL database were returning different than PGRES_COMMAND_OK, hence > > triggering the reprocess mechanism. Thanks very much to Alan Turower for > > his support. > > ! fix, improved logging of elements received/sent at buffering point > > between > > core process and plugins. Also added explicit start/end purge log message > > for cases in which there is no data to purge. > > ! fix, signals.c: ignore_falling_child() now logs if a child process exited > > with abnormal conditions; this is useful to track writer processes > > (created > > by plugins) are terminated by a signal, ie. SEGV. This is already the > > case > > for plugins themselves, with the Core Process reporting a simlar log > > message in case of abnormal exit. Thanks very much to Rene Stoutjesdijk > > for his support. > > ! fix, preprocess-data.h: added supported functions minf, minb, minbpp and > > minppf to non SQL plugins. Thanks to Jared Deyo for reporting the issue. > > ! fix, nfprobe_plugin.c: IP protocol was not set up correctly for IPv6 > > traffic in NetFlow v9/IPFIX. Thanks to Gabriel Vermeulen his support > > solving the issue. > > > > > > NOTES. > > See UPGRADE file. > > > > > > Cheers, > > Paolo > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
