subject:"Re\: \[pmacct\-discussion\] 1.5.0rc3 nfacctd segfaults"

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-30 Thread Paolo Lucente

Hi,

Just a quick note for the archives, issue solved:

http://www.mail-archive.com/pmacct-commits@pmacct.net/msg01138.html

Cheers,
Paolo

On Wed, Jun 25, 2014 at 07:30:54AM +, Paolo Lucente wrote:
 Hi,
 
 I tried your query and a your setup in lab and all seem to behave
 good to me. Still i'd like to get to the bottom of this, if possible.
 Since it seems the problem is easy to reproduce on your box, would
 you have a chance to follow-up privately so that i can troubleshoot
 the issue on your box?
 
 Cheers,
 Paolo
 
 On Tue, Jun 24, 2014 at 09:43:02AM -0700, Tim Jackson wrote:
  It doesn't actually appear to be clearing the statistics that cause
  the memory to balloon.. I've started clearing both imt tables I have
  setup every 2 minutes and:
  
  # ps aux | grep -e 'USER\|nfacct'
  USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
  root  1512  0.0  6.7 208380 129404 ?   Ss   08:10   0:02
  nfacctd: Core Process [default]
  root  1514  0.0  6.7 210916 130180 ?   S08:10   0:02
  nfacctd: Tee Plugin [fanout]
  root  1527  0.0  6.8 211172 132232 ?   Ss   08:10   0:02
  nfacctd: Core Process [default]
  root  1529  0.0  7.4 221128 142364 ?   S08:10   0:03
  nfacctd: PostgreSQL Plugin [as]
  root  1554  0.1 13.5 340128 261184 ?   Ss   08:10   0:05
  nfacctd: Core Process [default]
  root  1556  0.3 10.5 282840 203400 ?   S08:10   0:12
  nfacctd: IMT Plugin [full]
  root  1557  0.2 27.5 608480 529064 ?   S08:10   0:10
  nfacctd: IMT Plugin [dst]
  root  2740  0.0  0.0 103256   816 pts/0R+   09:17   0:00 grep
  -e USER\|nfacct
  
  # ps aux | grep -e 'USER\|nfacct'
  USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
  root  1512  0.0  6.7 208380 129384 ?   Ss   08:10   0:03
  nfacctd: Core Process [default]
  root  1514  0.0  6.7 210916 130160 ?   S08:10   0:02
  nfacctd: Tee Plugin [fanout]
  root  1527  0.0  6.8 211172 132212 ?   Ss   08:10   0:04
  nfacctd: Core Process [default]
  root  1529  0.0  7.4 221260 142452 ?   S08:10   0:04
  nfacctd: PostgreSQL Plugin [as]
  root  1554  0.1 13.5 340128 261164 ?   Ss   08:10   0:07
  nfacctd: Core Process [default]
  root  1556  0.3 10.9 288932 209520 ?   S08:10   0:18
  nfacctd: IMT Plugin [full]
  root  1557  0.2 35.7 765692 686324 ?   S08:10   0:14
  nfacctd: IMT Plugin [dst]
  root  3114  0.3  0.8 222716 16044 ?S09:40   0:00
  nfacctd: pgsql Plugin -- DB Writer [as]
  root  3160  0.0  0.0 103256   816 pts/0R+   09:42   0:00 grep
  -e USER\|nfacct
  
  
  Is it possible my query is causing this?
  
  This runs every 2 minutes:
  
  pmacct -p /tmp/nfacctd-dst.pipe -l -O json -a -c tag2 -M 2;3 -T
  packets,1000
  
  Configuration:
  
  daemonize: true
  nfacctd_port: 5680
  plugins: memory[full], memory[dst]
  
  aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as,
  src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm,
  src_ext_comm
  aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host
  
  imt_path[full]: /tmp/nfacctd-full.pipe
  imt_path[dst]: /tmp/nfacctd-dst.pipe
  
  pre_tag_map: /opt/pmacct/etc/pretag.map
  
  ! Not sure if needed
  nfacctd_time_new: true
  !
  nfacctd_renormalize: true
  
  plugin_pipe_size: 131072000
  plugin_buffer_size: 6400
  imt_buckets: 65537
  imt_mem_pools_size: 1024000
  imt_mem_pools_number: 160
  
  
  On Mon, Jun 23, 2014 at 4:13 PM, Paolo Lucente pa...@pmacct.net wrote:
   Hi,
  
   Can you then verify/confirm if it's the clearing of the statistics
   generating the issue? Determining how to reproduce the issue would
   help a lot to quickly solve the bug.
  
   Cheers,
   Paolo
 
 ___
 pmacct-discussion mailing list
 http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-24 Thread Tim Jackson

It doesn't actually appear to be clearing the statistics that cause
the memory to balloon.. I've started clearing both imt tables I have
setup every 2 minutes and:

# ps aux | grep -e 'USER\|nfacct'
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  1512  0.0  6.7 208380 129404 ?   Ss   08:10   0:02
nfacctd: Core Process [default]
root  1514  0.0  6.7 210916 130180 ?   S08:10   0:02
nfacctd: Tee Plugin [fanout]
root  1527  0.0  6.8 211172 132232 ?   Ss   08:10   0:02
nfacctd: Core Process [default]
root  1529  0.0  7.4 221128 142364 ?   S08:10   0:03
nfacctd: PostgreSQL Plugin [as]
root  1554  0.1 13.5 340128 261184 ?   Ss   08:10   0:05
nfacctd: Core Process [default]
root  1556  0.3 10.5 282840 203400 ?   S08:10   0:12
nfacctd: IMT Plugin [full]
root  1557  0.2 27.5 608480 529064 ?   S08:10   0:10
nfacctd: IMT Plugin [dst]
root  2740  0.0  0.0 103256   816 pts/0R+   09:17   0:00 grep
-e USER\|nfacct

# ps aux | grep -e 'USER\|nfacct'
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  1512  0.0  6.7 208380 129384 ?   Ss   08:10   0:03
nfacctd: Core Process [default]
root  1514  0.0  6.7 210916 130160 ?   S08:10   0:02
nfacctd: Tee Plugin [fanout]
root  1527  0.0  6.8 211172 132212 ?   Ss   08:10   0:04
nfacctd: Core Process [default]
root  1529  0.0  7.4 221260 142452 ?   S08:10   0:04
nfacctd: PostgreSQL Plugin [as]
root  1554  0.1 13.5 340128 261164 ?   Ss   08:10   0:07
nfacctd: Core Process [default]
root  1556  0.3 10.9 288932 209520 ?   S08:10   0:18
nfacctd: IMT Plugin [full]
root  1557  0.2 35.7 765692 686324 ?   S08:10   0:14
nfacctd: IMT Plugin [dst]
root  3114  0.3  0.8 222716 16044 ?S09:40   0:00
nfacctd: pgsql Plugin -- DB Writer [as]
root  3160  0.0  0.0 103256   816 pts/0R+   09:42   0:00 grep
-e USER\|nfacct


Is it possible my query is causing this?

This runs every 2 minutes:

pmacct -p /tmp/nfacctd-dst.pipe -l -O json -a -c tag2 -M 2;3 -T
packets,1000

Configuration:

daemonize: true
nfacctd_port: 5680
plugins: memory[full], memory[dst]

aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as,
src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm,
src_ext_comm
aggregate[dst]: tag, tag2, in_iface, dst_as, dst_host

imt_path[full]: /tmp/nfacctd-full.pipe
imt_path[dst]: /tmp/nfacctd-dst.pipe

pre_tag_map: /opt/pmacct/etc/pretag.map

! Not sure if needed
nfacctd_time_new: true
!
nfacctd_renormalize: true

plugin_pipe_size: 131072000
plugin_buffer_size: 6400
imt_buckets: 65537
imt_mem_pools_size: 1024000
imt_mem_pools_number: 160


On Mon, Jun 23, 2014 at 4:13 PM, Paolo Lucente pa...@pmacct.net wrote:
 Hi,

 Can you then verify/confirm if it's the clearing of the statistics
 generating the issue? Determining how to reproduce the issue would
 help a lot to quickly solve the bug.

 Cheers,
 Paolo

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-23 Thread Tim Jackson

It looks like the IMT of the one I keep clearing the statistics on is
balooning.. Starts around 200-300mb then climbs up..

USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  4516  0.0  6.4 208372 123172 ?   Ss   Jun19   3:03
nfacctd: Core Process [default]
root  4518  0.0  6.4 210908 123264 ?   SJun19   2:40
nfacctd: Tee Plugin [fanout]
root  4553  0.0  6.5 211168 125608 ?   Ss   Jun19   3:21
nfacctd: Core Process [default]
root  4555  0.0  7.1 221392 137116 ?   SJun19   2:51
nfacctd: PostgreSQL Plugin [as]
root 10522  0.1  5.0 340124 96760 ?Ss   11:04   0:09
nfacctd: Core Process [default]
root 10524  0.2  9.2 302656 176924 ?   S11:04   0:13
nfacctd: IMT Plugin [full]
root 10525  0.3 34.1 854392 656748 ?   S11:04   0:17
nfacctd: IMT Plugin [dst]
root 12282  0.0  0.0 103256   832 pts/1S+   12:38   0:00 grep
-e USER\|nfacct

USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  4516  0.0  6.6 208372 128152 ?   Ss   Jun19   3:03
nfacctd: Core Process [default]
root  4518  0.0  6.6 210908 128248 ?   SJun19   2:40
nfacctd: Tee Plugin [fanout]
root  4553  0.0  6.6 211168 128440 ?   Ss   Jun19   3:21
nfacctd: Core Process [default]
root  4555  0.0  7.2 221392 139928 ?   SJun19   2:52
nfacctd: PostgreSQL Plugin [as]
root 10522  0.1 10.5 340124 203344 ?   Ss   11:04   0:10
nfacctd: Core Process [default]
root 10524  0.2 11.6 302656 222992 ?   S11:04   0:13
nfacctd: IMT Plugin [full]
root 10525  0.3 38.9 885676 748416 ?   S11:04   0:18
nfacctd: IMT Plugin [dst]
root 12306  0.2  0.7 222848 13896 ?S12:39   0:00
nfacctd: pgsql Plugin -- DB Writer [as]
root 12362  0.0  0.0 103252   784 pts/1D+   12:42   0:00 grep
-e USER\|nfacct

USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root  4516  0.0  6.1 208372 119088 ?   Ss   Jun19   3:03
nfacctd: Core Process [default]
root  4518  0.0  6.2 210908 119184 ?   SJun19   2:40
nfacctd: Tee Plugin [fanout]
root  4553  0.0  6.6 211168 127832 ?   Ss   Jun19   3:21
nfacctd: Core Process [default]
root  4555  0.0  7.2 221392 139260 ?   SJun19   2:52
nfacctd: PostgreSQL Plugin [as]
root 10522  0.1 10.8 340124 208884 ?   Ss   11:04   0:10
nfacctd: Core Process [default]
root 10524  0.2 11.0 302656 211920 ?   S11:04   0:13
nfacctd: IMT Plugin [full]
root 10525  0.3 40.0 901516 769124 ?   S11:04   0:19
nfacctd: IMT Plugin [dst]
root 12401  0.0  0.0 103252   652 pts/1D+   12:45   0:00 grep
-e USER\|nfacct


The [full] IMT is never cleared, and doesn't seem to exhibit this
behavior... I'm performing the queries in this instance with a lock
now as well.

On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente pa...@pmacct.net wrote:
 Hi Tim,

 Can you please track down memory utilization to see if it could
 be something related to that? Also, can you try performing a query
 with lock:

 shell pmacct -l  .. parameters .. 

 If none of this helps, then yes, proceed to capture segfault data
 with gdb.

 Cheers,
 Paolo

 On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote:
 We're having some issues using nfacctd with IMT.. After running for
 ~6-8 hours ingesting flow data, we see segfaults and the pmacct client
 ceases to function properly returning:

 ERROR: missing EOF from server

 Querying pmacct client every 2 minutes with:

 pmacct -p nfacctd-dst.pipe -O json -a -c tag2 -M 2;3 -T packets,1000

 If that returns data, we then:

 pmacct -p nfacctd-dst.pipe -e

 Associated segfault from nfacctd daemon:

 Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip
 0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
 Jun 20 11:32:02 kernel: nfacctd[22903]: segfault at 4d8e6600 ip
 00476103 sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]

 nfacctd Config:

 !daemonize: true
 nfacctd_port: 5680
 plugins: memory[full], memory[dst]

 aggregate[full]: tag, tag2, in_iface, out_iface, src_as, dst_as,
 src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm,

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-23 Thread Paolo Lucente

Hi,

Can you then verify/confirm if it's the clearing of the statistics
generating the issue? Determining how to reproduce the issue would
help a lot to quickly solve the bug.

Cheers,
Paolo

On Mon, Jun 23, 2014 at 12:47:19PM -0700, Tim Jackson wrote:
 It looks like the IMT of the one I keep clearing the statistics on is
 balooning.. Starts around 200-300mb then climbs up..
 
 USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
 root  4516  0.0  6.4 208372 123172 ?   Ss   Jun19   3:03
 nfacctd: Core Process [default]
 root  4518  0.0  6.4 210908 123264 ?   SJun19   2:40
 nfacctd: Tee Plugin [fanout]
 root  4553  0.0  6.5 211168 125608 ?   Ss   Jun19   3:21
 nfacctd: Core Process [default]
 root  4555  0.0  7.1 221392 137116 ?   SJun19   2:51
 nfacctd: PostgreSQL Plugin [as]
 root 10522  0.1  5.0 340124 96760 ?Ss   11:04   0:09
 nfacctd: Core Process [default]
 root 10524  0.2  9.2 302656 176924 ?   S11:04   0:13
 nfacctd: IMT Plugin [full]
 root 10525  0.3 34.1 854392 656748 ?   S11:04   0:17
 nfacctd: IMT Plugin [dst]
 root 12282  0.0  0.0 103256   832 pts/1S+   12:38   0:00 grep
 -e USER\|nfacct
 
 USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
 root  4516  0.0  6.6 208372 128152 ?   Ss   Jun19   3:03
 nfacctd: Core Process [default]
 root  4518  0.0  6.6 210908 128248 ?   SJun19   2:40
 nfacctd: Tee Plugin [fanout]
 root  4553  0.0  6.6 211168 128440 ?   Ss   Jun19   3:21
 nfacctd: Core Process [default]
 root  4555  0.0  7.2 221392 139928 ?   SJun19   2:52
 nfacctd: PostgreSQL Plugin [as]
 root 10522  0.1 10.5 340124 203344 ?   Ss   11:04   0:10
 nfacctd: Core Process [default]
 root 10524  0.2 11.6 302656 222992 ?   S11:04   0:13
 nfacctd: IMT Plugin [full]
 root 10525  0.3 38.9 885676 748416 ?   S11:04   0:18
 nfacctd: IMT Plugin [dst]
 root 12306  0.2  0.7 222848 13896 ?S12:39   0:00
 nfacctd: pgsql Plugin -- DB Writer [as]
 root 12362  0.0  0.0 103252   784 pts/1D+   12:42   0:00 grep
 -e USER\|nfacct
 
 USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
 root  4516  0.0  6.1 208372 119088 ?   Ss   Jun19   3:03
 nfacctd: Core Process [default]
 root  4518  0.0  6.2 210908 119184 ?   SJun19   2:40
 nfacctd: Tee Plugin [fanout]
 root  4553  0.0  6.6 211168 127832 ?   Ss   Jun19   3:21
 nfacctd: Core Process [default]
 root  4555  0.0  7.2 221392 139260 ?   SJun19   2:52
 nfacctd: PostgreSQL Plugin [as]
 root 10522  0.1 10.8 340124 208884 ?   Ss   11:04   0:10
 nfacctd: Core Process [default]
 root 10524  0.2 11.0 302656 211920 ?   S11:04   0:13
 nfacctd: IMT Plugin [full]
 root 10525  0.3 40.0 901516 769124 ?   S11:04   0:19
 nfacctd: IMT Plugin [dst]
 root 12401  0.0  0.0 103252   652 pts/1D+   12:45   0:00 grep
 -e USER\|nfacct
 
 
 The [full] IMT is never cleared, and doesn't seem to exhibit this
 behavior... I'm performing the queries in this instance with a lock
 now as well.
 
 On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente pa...@pmacct.net wrote:
  Hi Tim,
 
  Can you please track down memory utilization to see if it could
  be something related to that? Also, can you try performing a query
  with lock:
 
  shell pmacct -l  .. parameters .. 
 
  If none of this helps, then yes, proceed to capture segfault data
  with gdb.
 
  Cheers,
  Paolo
 
  On Fri, Jun 20, 2014 at 11:45:57AM -0700, Tim Jackson wrote:
  We're having some issues using nfacctd with IMT.. After running for
  ~6-8 hours ingesting flow data, we see segfaults and the pmacct client
  ceases to function properly returning:
 
  ERROR: missing EOF from server
 
  Querying pmacct client every 2 minutes with:
 
  pmacct -p nfacctd-dst.pipe -O json -a -c tag2 -M 2;3 -T packets,1000
 
  If that returns data, we then:
 
  pmacct -p nfacctd-dst.pipe -e
 
  Associated segfault from nfacctd daemon:
 
  Jun 20 10:32:02 kernel: nfacctd[21874]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 10:36:02 kernel: nfacctd[21930]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 10:40:02 kernel: nfacctd[21983]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 10:46:02 kernel: nfacctd[22068]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 10:54:02 kernel: nfacctd[22188]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 11:02:02 kernel: nfacctd[22350]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20 11:04:02 kernel: nfacctd[22374]: segfault at 21 ip
  0047613d sp 7fff9ad3e1b0 error 4 in nfacctd[40+de000]
  Jun 20

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

4 matches

Site Navigation

Mail list logo

Footer information