Hi, Can you then verify/confirm if it's the clearing of the statistics generating the issue? Determining how to reproduce the issue would help a lot to quickly solve the bug.
Cheers, Paolo On Mon, Jun 23, 2014 at 12:47:19PM -0700, Tim Jackson wrote: > It looks like the IMT of the one I keep clearing the statistics on is > balooning.. Starts around 200-300mb then climbs up.. > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 4516 0.0 6.4 208372 123172 ? Ss Jun19 3:03 > nfacctd: Core Process [default] > root 4518 0.0 6.4 210908 123264 ? S Jun19 2:40 > nfacctd: Tee Plugin [fanout] > root 4553 0.0 6.5 211168 125608 ? Ss Jun19 3:21 > nfacctd: Core Process [default] > root 4555 0.0 7.1 221392 137116 ? S Jun19 2:51 > nfacctd: PostgreSQL Plugin [as] > root 10522 0.1 5.0 340124 96760 ? Ss 11:04 0:09 > nfacctd: Core Process [default] > root 10524 0.2 9.2 302656 176924 ? S 11:04 0:13 > nfacctd: IMT Plugin [full] > root 10525 0.3 34.1 854392 656748 ? S 11:04 0:17 > nfacctd: IMT Plugin [dst] > root 12282 0.0 0.0 103256 832 pts/1 S+ 12:38 0:00 grep > -e USER\|nfacct > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 4516 0.0 6.6 208372 128152 ? Ss Jun19 3:03 > nfacctd: Core Process [default] > root 4518 0.0 6.6 210908 128248 ? S Jun19 2:40 > nfacctd: Tee Plugin [fanout] > root 4553 0.0 6.6 211168 128440 ? Ss Jun19 3:21 > nfacctd: Core Process [default] > root 4555 0.0 7.2 221392 139928 ? S Jun19 2:52 > nfacctd: PostgreSQL Plugin [as] > root 10522 0.1 10.5 340124 203344 ? Ss 11:04 0:10 > nfacctd: Core Process [default] > root 10524 0.2 11.6 302656 222992 ? S 11:04 0:13 > nfacctd: IMT Plugin [full] > root 10525 0.3 38.9 885676 748416 ? S 11:04 0:18 > nfacctd: IMT Plugin [dst] > root 12306 0.2 0.7 222848 13896 ? S 12:39 0:00 > nfacctd: pgsql Plugin -- DB Writer [as] > root 12362 0.0 0.0 103252 784 pts/1 D+ 12:42 0:00 grep > -e USER\|nfacct > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 4516 0.0 6.1 208372 119088 ? Ss Jun19 3:03 > nfacctd: Core Process [default] > root 4518 0.0 6.2 210908 119184 ? S Jun19 2:40 > nfacctd: Tee Plugin [fanout] > root 4553 0.0 6.6 211168 127832 ? Ss Jun19 3:21 > nfacctd: Core Process [default] > root 4555 0.0 7.2 221392 139260 ? S Jun19 2:52 > nfacctd: PostgreSQL Plugin [as] > root 10522 0.1 10.8 340124 208884 ? Ss 11:04 0:10 > nfacctd: Core Process [default] > root 10524 0.2 11.0 302656 211920 ? S 11:04 0:13 > nfacctd: IMT Plugin [full] > root 10525 0.3 40.0 901516 769124 ? S 11:04 0:19 > nfacctd: IMT Plugin [dst] > root 12401 0.0 0.0 103252 652 pts/1 D+ 12:45 0:00 grep > -e USER\|nfacct > > > The [full] IMT is never cleared, and doesn't seem to exhibit this > behavior... I'm performing the queries in this instance with a lock > now as well. > > On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente <[email protected]> wrote: > > Hi Tim, > > > > Can you please track down memory utilization to see if it could > > be something related to that? Also, can you try performing a query > > with lock: > > > > shell> pmacct -l < .. parameters .. > > > > > If none of this helps, then yes, proceed to capture segfault data > > with gdb. > > > > Cheers, > > Paolo > > > > On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote: > >> We're having some issues using nfacctd with IMT.. After running for > >> ~6-8 hours ingesting flow data, we see segfaults and the pmacct client > >> ceases to function properly returning: > >> > >> "ERROR: missing EOF from server" > >> > >> Querying pmacct client every 2 minutes with: > >> > >> pmacct -p nfacctd-dst.pipe -O json -a -c "tag2" -M "2;3" -T "packets,1000" > >> > >> If that returns data, we then: > >> > >> pmacct -p nfacctd-dst.pipe -e > >> > >> Associated segfault from nfacctd daemon: > >> > >> Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip > >> 000000000047613d sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> Jun 20 11:32:02 kernel: nfacctd[22903]: segfault at 4d8e6600 ip > >> 0000000000476103 sp 00007fff9ad3e1b0 error 4 in nfacctd[400000+de000] > >> > >> nfacctd Config: > >> > >> !daemonize: true > >> nfacctd_port: 5680 > >> plugins: memory[full], memory[dst] > >> > >> aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, > >> src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, > >> src_ext_comm > >> aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host > >> > >> imt_path[full]: /tmp/nfacctd-full.pipe > >> imt_path[dst]: /tmp/nfacctd-dst.pipe > >> > >> pre_tag_map: /opt/pmacct/etc/pretag.map > >> > >> nfacctd_time_new: true > >> nfacctd_renormalize: true > >> > >> plugin_pipe_size: 131072000 > >> plugin_buffer_size: 6400 > >> imt_buckets: 65537 > >> imt_mem_pools_size: 1024000 > >> > >> I'm working on capturing the debug output from nfacctd when this > >> segfault happens, but is there anything else I should capture to help > >> figure out why this is happening? > >> > >> _______________________________________________ > >> pmacct-discussion mailing list > >> http://www.pmacct.net/#mailinglists > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
