It looks like the IMT of the one I keep clearing the statistics on is balooning.. Starts around 200-300mb then climbs up..
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.4 208372 123172 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.4 210908 123264 ? S Jun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.5 211168 125608 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.1 221392 137116 ? S Jun19 2:51 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 5.0 340124 96760 ? Ss 11:04 0:09 nfacctd: Core Process [default] root 10524 0.2 9.2 302656 176924 ? S 11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 34.1 854392 656748 ? S 11:04 0:17 nfacctd: IMT Plugin [dst] root 12282 0.0 0.0 103256 832 pts/1 S+ 12:38 0:00 grep -e USER\|nfacct USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.6 208372 128152 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.6 210908 128248 ? S Jun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 128440 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139928 ? S Jun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.5 340124 203344 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.6 302656 222992 ? S 11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 38.9 885676 748416 ? S 11:04 0:18 nfacctd: IMT Plugin [dst] root 12306 0.2 0.7 222848 13896 ? S 12:39 0:00 nfacctd: pgsql Plugin -- DB Writer [as] root 12362 0.0 0.0 103252 784 pts/1 D+ 12:42 0:00 grep -e USER\|nfacct USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.1 208372 119088 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.2 210908 119184 ? S Jun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 127832 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139260 ? S Jun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.8 340124 208884 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.0 302656 211920 ? S 11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 40.0 901516 769124 ? S 11:04 0:19 nfacctd: IMT Plugin [dst] root 12401 0.0 0.0 103252 652 pts/1 D+ 12:45 0:00 grep -e USER\|nfacct The [full] IMT is never cleared, and doesn't seem to exhibit this behavior... I'm performing the queries in this instance with a lock now as well. On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente <[email protected]> wrote: > Hi Tim, > > Can you please track down memory utilization to see if it could > be something related to that? Also, can you try performing a query > with lock: > > shell> pmacct -l < .. parameters .. > > > If none of this helps, then yes, proceed to capture segfault data > with gdb. > > Cheers, > Paolo > > On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote: >> We're having some issues using nfacctd with IMT.. After running for >> ~6-8 hours ingesting flow data, we see segfaults and the pmacct client >> ceases to function properly returning: >> >> "ERROR: missing EOF from server" >> >> Querying pmacct client every 2 minutes with: >> >> pmacct -p nfacctd-dst.pipe -O json -a -c "tag2" -M "2;3" -T "packets,1000" >> >> If that returns data, we then: >> >> pmacct -p nfacctd-dst.pipe -e >> >> Associated segfault from nfacctd daemon: >> >> Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> Jun 20 11:32:02 kernel: nfacctd[22903]: segfault at 4d8e6600 ip >> 0000000000476103 sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] >> >> nfacctd Config: >> >> !daemonize: true >> nfacctd_port: 5680 >> plugins: memory[full], memory[dst] >> >> aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, >> src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, >> src_ext_comm >> aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host >> >> imt_path[full]: /tmp/nfacctd-full.pipe >> imt_path[dst]: /tmp/nfacctd-dst.pipe >> >> pre_tag_map: /opt/pmacct/etc/pretag.map >> >> nfacctd_time_new: true >> nfacctd_renormalize: true >> >> plugin_pipe_size: 131072000 >> plugin_buffer_size: 6400 >> imt_buckets: 65537 >> imt_mem_pools_size: 1024000 >> >> I'm working on capturing the debug output from nfacctd when this >> segfault happens, but is there anything else I should capture to help >> figure out why this is happening? >> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
