Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults
Hi, Just a quick note for the archives, issue solved: http://www.mail-archive.com/pmacct-commits@pmacct.net/msg01138.html Cheers, Paolo On Wed, Jun 25, 2014 at 07:30:54AM +, Paolo Lucente wrote: Hi, I tried your query and a your setup in lab and all seem to behave good to me. Still i'd like to get to the bottom of this, if possible. Since it seems the problem is easy to reproduce on your box, would you have a chance to follow-up privately so that i can troubleshoot the issue on your box? Cheers, Paolo On Tue, Jun 24, 2014 at 09:43:02AM -0700, Tim Jackson wrote: It doesn't actually appear to be clearing the statistics that cause the memory to balloon.. I've started clearing both imt tables I have setup every 2 minutes and: # ps aux | grep -e 'USER\|nfacct' USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 1512 0.0 6.7 208380 129404 ? Ss 08:10 0:02 nfacctd: Core Process [default] root 1514 0.0 6.7 210916 130180 ? S08:10 0:02 nfacctd: Tee Plugin [fanout] root 1527 0.0 6.8 211172 132232 ? Ss 08:10 0:02 nfacctd: Core Process [default] root 1529 0.0 7.4 221128 142364 ? S08:10 0:03 nfacctd: PostgreSQL Plugin [as] root 1554 0.1 13.5 340128 261184 ? Ss 08:10 0:05 nfacctd: Core Process [default] root 1556 0.3 10.5 282840 203400 ? S08:10 0:12 nfacctd: IMT Plugin [full] root 1557 0.2 27.5 608480 529064 ? S08:10 0:10 nfacctd: IMT Plugin [dst] root 2740 0.0 0.0 103256 816 pts/0R+ 09:17 0:00 grep -e USER\|nfacct # ps aux | grep -e 'USER\|nfacct' USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 1512 0.0 6.7 208380 129384 ? Ss 08:10 0:03 nfacctd: Core Process [default] root 1514 0.0 6.7 210916 130160 ? S08:10 0:02 nfacctd: Tee Plugin [fanout] root 1527 0.0 6.8 211172 132212 ? Ss 08:10 0:04 nfacctd: Core Process [default] root 1529 0.0 7.4 221260 142452 ? S08:10 0:04 nfacctd: PostgreSQL Plugin [as] root 1554 0.1 13.5 340128 261164 ? Ss 08:10 0:07 nfacctd: Core Process [default] root 1556 0.3 10.9 288932 209520 ? S08:10 0:18 nfacctd: IMT Plugin [full] root 1557 0.2 35.7 765692 686324 ? S08:10 0:14 nfacctd: IMT Plugin [dst] root 3114 0.3 0.8 222716 16044 ?S09:40 0:00 nfacctd: pgsql Plugin -- DB Writer [as] root 3160 0.0 0.0 103256 816 pts/0R+ 09:42 0:00 grep -e USER\|nfacct Is it possible my query is causing this? This runs every 2 minutes: pmacct -p /tmp/nfacctd-dst.pipe -l -O json -a -c tag2 -M 2;3 -T packets,1000 Configuration: daemonize: true nfacctd_port: 5680 plugins: memory[full], memory[dst] aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, src_ext_comm aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host imt_path[full]: /tmp/nfacctd-full.pipe imt_path[dst]: /tmp/nfacctd-dst.pipe pre_tag_map: /opt/pmacct/etc/pretag.map ! Not sure if needed nfacctd_time_new: true ! nfacctd_renormalize: true plugin_pipe_size: 131072000 plugin_buffer_size: 6400 imt_buckets: 65537 imt_mem_pools_size: 1024000 imt_mem_pools_number: 160 On Mon, Jun 23, 2014 at 4:13 PM, Paolo Lucente pa...@pmacct.net wrote: Hi, Can you then verify/confirm if it's the clearing of the statistics generating the issue? Determining how to reproduce the issue would help a lot to quickly solve the bug. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults
It doesn't actually appear to be clearing the statistics that cause the memory to balloon.. I've started clearing both imt tables I have setup every 2 minutes and: # ps aux | grep -e 'USER\|nfacct' USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 1512 0.0 6.7 208380 129404 ? Ss 08:10 0:02 nfacctd: Core Process [default] root 1514 0.0 6.7 210916 130180 ? S08:10 0:02 nfacctd: Tee Plugin [fanout] root 1527 0.0 6.8 211172 132232 ? Ss 08:10 0:02 nfacctd: Core Process [default] root 1529 0.0 7.4 221128 142364 ? S08:10 0:03 nfacctd: PostgreSQL Plugin [as] root 1554 0.1 13.5 340128 261184 ? Ss 08:10 0:05 nfacctd: Core Process [default] root 1556 0.3 10.5 282840 203400 ? S08:10 0:12 nfacctd: IMT Plugin [full] root 1557 0.2 27.5 608480 529064 ? S08:10 0:10 nfacctd: IMT Plugin [dst] root 2740 0.0 0.0 103256 816 pts/0R+ 09:17 0:00 grep -e USER\|nfacct # ps aux | grep -e 'USER\|nfacct' USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 1512 0.0 6.7 208380 129384 ? Ss 08:10 0:03 nfacctd: Core Process [default] root 1514 0.0 6.7 210916 130160 ? S08:10 0:02 nfacctd: Tee Plugin [fanout] root 1527 0.0 6.8 211172 132212 ? Ss 08:10 0:04 nfacctd: Core Process [default] root 1529 0.0 7.4 221260 142452 ? S08:10 0:04 nfacctd: PostgreSQL Plugin [as] root 1554 0.1 13.5 340128 261164 ? Ss 08:10 0:07 nfacctd: Core Process [default] root 1556 0.3 10.9 288932 209520 ? S08:10 0:18 nfacctd: IMT Plugin [full] root 1557 0.2 35.7 765692 686324 ? S08:10 0:14 nfacctd: IMT Plugin [dst] root 3114 0.3 0.8 222716 16044 ?S09:40 0:00 nfacctd: pgsql Plugin -- DB Writer [as] root 3160 0.0 0.0 103256 816 pts/0R+ 09:42 0:00 grep -e USER\|nfacct Is it possible my query is causing this? This runs every 2 minutes: pmacct -p /tmp/nfacctd-dst.pipe -l -O json -a -c tag2 -M 2;3 -T packets,1000 Configuration: daemonize: true nfacctd_port: 5680 plugins: memory[full], memory[dst] aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, src_ext_comm aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host imt_path[full]: /tmp/nfacctd-full.pipe imt_path[dst]: /tmp/nfacctd-dst.pipe pre_tag_map: /opt/pmacct/etc/pretag.map ! Not sure if needed nfacctd_time_new: true ! nfacctd_renormalize: true plugin_pipe_size: 131072000 plugin_buffer_size: 6400 imt_buckets: 65537 imt_mem_pools_size: 1024000 imt_mem_pools_number: 160 On Mon, Jun 23, 2014 at 4:13 PM, Paolo Lucente pa...@pmacct.net wrote: Hi, Can you then verify/confirm if it's the clearing of the statistics generating the issue? Determining how to reproduce the issue would help a lot to quickly solve the bug. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults
It looks like the IMT of the one I keep clearing the statistics on is balooning.. Starts around 200-300mb then climbs up.. USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.4 208372 123172 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.4 210908 123264 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.5 211168 125608 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.1 221392 137116 ? SJun19 2:51 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 5.0 340124 96760 ?Ss 11:04 0:09 nfacctd: Core Process [default] root 10524 0.2 9.2 302656 176924 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 34.1 854392 656748 ? S11:04 0:17 nfacctd: IMT Plugin [dst] root 12282 0.0 0.0 103256 832 pts/1S+ 12:38 0:00 grep -e USER\|nfacct USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.6 208372 128152 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.6 210908 128248 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 128440 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139928 ? SJun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.5 340124 203344 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.6 302656 222992 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 38.9 885676 748416 ? S11:04 0:18 nfacctd: IMT Plugin [dst] root 12306 0.2 0.7 222848 13896 ?S12:39 0:00 nfacctd: pgsql Plugin -- DB Writer [as] root 12362 0.0 0.0 103252 784 pts/1D+ 12:42 0:00 grep -e USER\|nfacct USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.1 208372 119088 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.2 210908 119184 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 127832 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139260 ? SJun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.8 340124 208884 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.0 302656 211920 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 40.0 901516 769124 ? S11:04 0:19 nfacctd: IMT Plugin [dst] root 12401 0.0 0.0 103252 652 pts/1D+ 12:45 0:00 grep -e USER\|nfacct The [full] IMT is never cleared, and doesn't seem to exhibit this behavior... I'm performing the queries in this instance with a lock now as well. On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente pa...@pmacct.net wrote: Hi Tim, Can you please track down memory utilization to see if it could be something related to that? Also, can you try performing a query with lock: shell pmacct -l .. parameters .. If none of this helps, then yes, proceed to capture segfault data with gdb. Cheers, Paolo On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote: We're having some issues using nfacctd with IMT.. After running for ~6-8 hours ingesting flow data, we see segfaults and the pmacct client ceases to function properly returning: ERROR: missing EOF from server Querying pmacct client every 2 minutes with: pmacct -p nfacctd-dst.pipe -O json -a -c tag2 -M 2;3 -T packets,1000 If that returns data, we then: pmacct -p nfacctd-dst.pipe -e Associated segfault from nfacctd daemon: Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:32:02 kernel: nfacctd[22903]: segfault at 4d8e6600 ip 00476103 sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] nfacctd Config: !daemonize: true nfacctd_port: 5680 plugins: memory[full], memory[dst] aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm,
Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults
Hi, Can you then verify/confirm if it's the clearing of the statistics generating the issue? Determining how to reproduce the issue would help a lot to quickly solve the bug. Cheers, Paolo On Mon, Jun 23, 2014 at 12:47:19PM -0700, Tim Jackson wrote: It looks like the IMT of the one I keep clearing the statistics on is balooning.. Starts around 200-300mb then climbs up.. USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.4 208372 123172 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.4 210908 123264 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.5 211168 125608 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.1 221392 137116 ? SJun19 2:51 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 5.0 340124 96760 ?Ss 11:04 0:09 nfacctd: Core Process [default] root 10524 0.2 9.2 302656 176924 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 34.1 854392 656748 ? S11:04 0:17 nfacctd: IMT Plugin [dst] root 12282 0.0 0.0 103256 832 pts/1S+ 12:38 0:00 grep -e USER\|nfacct USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.6 208372 128152 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.6 210908 128248 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 128440 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139928 ? SJun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.5 340124 203344 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.6 302656 222992 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 38.9 885676 748416 ? S11:04 0:18 nfacctd: IMT Plugin [dst] root 12306 0.2 0.7 222848 13896 ?S12:39 0:00 nfacctd: pgsql Plugin -- DB Writer [as] root 12362 0.0 0.0 103252 784 pts/1D+ 12:42 0:00 grep -e USER\|nfacct USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 4516 0.0 6.1 208372 119088 ? Ss Jun19 3:03 nfacctd: Core Process [default] root 4518 0.0 6.2 210908 119184 ? SJun19 2:40 nfacctd: Tee Plugin [fanout] root 4553 0.0 6.6 211168 127832 ? Ss Jun19 3:21 nfacctd: Core Process [default] root 4555 0.0 7.2 221392 139260 ? SJun19 2:52 nfacctd: PostgreSQL Plugin [as] root 10522 0.1 10.8 340124 208884 ? Ss 11:04 0:10 nfacctd: Core Process [default] root 10524 0.2 11.0 302656 211920 ? S11:04 0:13 nfacctd: IMT Plugin [full] root 10525 0.3 40.0 901516 769124 ? S11:04 0:19 nfacctd: IMT Plugin [dst] root 12401 0.0 0.0 103252 652 pts/1D+ 12:45 0:00 grep -e USER\|nfacct The [full] IMT is never cleared, and doesn't seem to exhibit this behavior... I'm performing the queries in this instance with a lock now as well. On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente pa...@pmacct.net wrote: Hi Tim, Can you please track down memory utilization to see if it could be something related to that? Also, can you try performing a query with lock: shell pmacct -l .. parameters .. If none of this helps, then yes, proceed to capture segfault data with gdb. Cheers, Paolo On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote: We're having some issues using nfacctd with IMT.. After running for ~6-8 hours ingesting flow data, we see segfaults and the pmacct client ceases to function properly returning: ERROR: missing EOF from server Querying pmacct client every 2 minutes with: pmacct -p nfacctd-dst.pipe -O json -a -c tag2 -M 2;3 -T packets,1000 If that returns data, we then: pmacct -p nfacctd-dst.pipe -e Associated segfault from nfacctd daemon: Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20
[pmacct-discussion] 1.5.0rc3 nfacctd segfaults
We're having some issues using nfacctd with IMT.. After running for ~6-8 hours ingesting flow data, we see segfaults and the pmacct client ceases to function properly returning: ERROR: missing EOF from server Querying pmacct client every 2 minutes with: pmacct -p nfacctd-dst.pipe -O json -a -c tag2 -M 2;3 -T packets,1000 If that returns data, we then: pmacct -p nfacctd-dst.pipe -e Associated segfault from nfacctd daemon: Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] Jun 20 11:32:02 kernel: nfacctd[22903]: segfault at 4d8e6600 ip 00476103 sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000] nfacctd Config: !daemonize: true nfacctd_port: 5680 plugins: memory[full], memory[dst] aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as, src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, src_ext_comm aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host imt_path[full]: /tmp/nfacctd-full.pipe imt_path[dst]: /tmp/nfacctd-dst.pipe pre_tag_map: /opt/pmacct/etc/pretag.map nfacctd_time_new: true nfacctd_renormalize: true plugin_pipe_size: 131072000 plugin_buffer_size: 6400 imt_buckets: 65537 imt_mem_pools_size: 1024000 I'm working on capturing the debug output from nfacctd when this segfault happens, but is there anything else I should capture to help figure out why this is happening? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists