Re: [pmacct-discussion] Off by one warning?
Paolo, Sorry I missed that you had replied. Yes, these happen all the time. There's a big burst on startup and then a pretty steady one afterwards. It looks like the later burst might be due to sending two streams? INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 6, ET: 0) *** WARN: expecting flow '25593234' but received '234608' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234609' but received '25593234' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593299' but received '234609' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234610' but received '25593299' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593367' but received '234610' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234611' but received '25593367' collector=0.0.0.0:6001agent=X:0 INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 7, ET: 0) *** WARN: expecting flow '25593429' but received '234611' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234612' but received '25593429' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593510' but received '234612' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234613' but received '25593510' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593572' but received '234613' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234614' but received '25593572' collector=0.0.0.0:6001agent=X:0 See how the flow numbers flip back and forth between 234k and 25M? I'm willing to disable checks, but I wouldn't want to miss other debug information in my testing. Cheers, Joel On Mon, Nov 11, 2013 at 3:16 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Joel, Could also be packets are received out of order, which can be harmless depending on the use-cases. Anyway if annoying these messages can be disabled by setting nfacctd_disable_checks to true. I propose this idea because i don't seem to have seen such warnings on a regular basis on other IPFIX exports. Maybe would help if you can define better frequently. Is that like in always, at times, in specific times of the day, or ..? Cheers, Paolo On Sun, Nov 10, 2013 at 06:26:22PM -0800, Joel Krauska wrote: (I should have mentioned I'm testing rc1 NetFlow Accounting Daemon, nfacctd 1.5.0rc1 (20130829-00) --enable-mysql --enable-64bit --enable-threads --enable-geoip I frequently get these Warnings. WARN: expecting flow '4423369' but received '4423371' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423371' but received '4423372' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423372' but received '4423374' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423374' but received '4423375' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423375' but received '4423376' collector=0.0.0.0:6001agent=BLAH:0 It seems odd to see them in series like this, since the 'expected' usually is the one it just received just before... Looks like possibly an off by 1 error? Cheers, Joel ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] buffer overflow / backtrace on 1.5rc1
I get a pretty repeatable buffer overflow when trying to use nfacctd with BGP enabled. (threaded) It will run for a few moments and then bombs out. *** buffer overflow detected ***: nfacctd: Core Process [default] terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f56d4dd1f47] /lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f56d4dd0e40] nfacctd: Core Process [default](bgp_nlri_parse+0x15f)[0x46b17f] nfacctd: Core Process [default](bgp_update_msg+0x3a7)[0x46bb87] nfacctd: Core Process [default](skinny_bgp_daemon+0xc1f)[0x46e84f] nfacctd: Core Process [default](thread_runner+0x5b)[0x45f72b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f56d508ee9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f56d4dbb3fd] === Memory map: 0040-004ca000 r-xp fd:01 270072 /opt/pmacct/sbin/nfacctd 006c9000-006ca000 r--p 000c9000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006ca000-006cb000 rw-p 000ca000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006cb000-0074c000 rw-p 00:00 0 00ab9000-00ada000 rw-p 00:00 0 [heap] 7f56c400-7f56c6e8c000 rw-p 00:00 0 7f56c6e8c000-7f56c800 ---p 00:00 0 7f56cbde2000-7f56cbdf7000 r-xp fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbdf7000-7f56cbff6000 ---p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff6000-7f56cbff7000 r--p 00014000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff7000-7f56cbff8000 rw-p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbfff000-7f56cc00 rw-p 00:00 0 7f56cc00-7f56d000 rw-p 00:00 0 7f56d3a0-7f56d3fb3000 rw-s 00:04 10398639 /dev/zero (deleted) 7f56d3fb3000-7f56d3fb4000 ---p 00:00 0 7f56d3fb4000-7f56d47b4000 rw-p 00:00 0 7f56d47b4000-7f56d48af000 r-xp fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d48af000-7f56d4aae000 ---p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aae000-7f56d4aaf000 r--p 000fa000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aaf000-7f56d4ab rw-p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4ab-7f56d4ac6000 r-xp fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4ac6000-7f56d4cc5000 ---p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc5000-7f56d4cc6000 r--p 00015000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc6000-7f56d4cc7000 rw-p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc7000-7f56d4e7c000 r-xp fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d4e7c000-7f56d507c000 ---p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d507c000-7f56d508 r--p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d508-7f56d5082000 rw-p 001b9000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d5082000-7f56d5087000 rw-p 00:00 0 7f56d5087000-7f56d509f000 r-xp fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d509f000-7f56d529e000 ---p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529e000-7f56d529f000 r--p 00017000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529f000-7f56d52a rw-p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d52a-7f56d52a4000 rw-p 00:00 0 7f56d52a4000-7f56d52a6000 r-xp fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d52a6000-7f56d54a6000 ---p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a6000-7f56d54a7000 r--p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a7000-7f56d54a8000 rw-p 3000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a8000-7f56d54db000 r-xp fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d54db000-7f56d56da000 ---p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56da000-7f56d56db000 r--p 00032000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56db000-7f56d56dd000 rw-p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56dd000-7f56d5711000 r-xp fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5711000-7f56d5911000 ---p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5911000-7f56d5912000 r--p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5912000-7f56d5913000 rw-p 00035000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5913000-7f56d5914000 rw-p 00:00 0 7f56d5914000-7f56d5bd2000 r-xp fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5bd2000-7f56d5dd1000 ---p 002be000 fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5dd1000-7f56d5dd7000 r--p 002bd000 fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5dd7000-7f56d5e55000 rw-p 002c3000 fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5e55000-7f56d5e5a000 rw-p 00:00 0 7f56d5e5a000-7f56d5e7c000 r-xp fd:01 394153
Re: [pmacct-discussion] buffer overflow / backtrace on 1.5rc1
FWIW: I get the same on nfacctd 0.14.3 (20130503-00) On Wed, Dec 4, 2013 at 4:18 PM, Joel Krauska j...@krauska.net wrote: I get a pretty repeatable buffer overflow when trying to use nfacctd with BGP enabled. (threaded) It will run for a few moments and then bombs out. *** buffer overflow detected ***: nfacctd: Core Process [default] terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f56d4dd1f47] /lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f56d4dd0e40] nfacctd: Core Process [default](bgp_nlri_parse+0x15f)[0x46b17f] nfacctd: Core Process [default](bgp_update_msg+0x3a7)[0x46bb87] nfacctd: Core Process [default](skinny_bgp_daemon+0xc1f)[0x46e84f] nfacctd: Core Process [default](thread_runner+0x5b)[0x45f72b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f56d508ee9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f56d4dbb3fd] === Memory map: 0040-004ca000 r-xp fd:01 270072 /opt/pmacct/sbin/nfacctd 006c9000-006ca000 r--p 000c9000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006ca000-006cb000 rw-p 000ca000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006cb000-0074c000 rw-p 00:00 0 00ab9000-00ada000 rw-p 00:00 0 [heap] 7f56c400-7f56c6e8c000 rw-p 00:00 0 7f56c6e8c000-7f56c800 ---p 00:00 0 7f56cbde2000-7f56cbdf7000 r-xp fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbdf7000-7f56cbff6000 ---p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff6000-7f56cbff7000 r--p 00014000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff7000-7f56cbff8000 rw-p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbfff000-7f56cc00 rw-p 00:00 0 7f56cc00-7f56d000 rw-p 00:00 0 7f56d3a0-7f56d3fb3000 rw-s 00:04 10398639 /dev/zero (deleted) 7f56d3fb3000-7f56d3fb4000 ---p 00:00 0 7f56d3fb4000-7f56d47b4000 rw-p 00:00 0 7f56d47b4000-7f56d48af000 r-xp fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d48af000-7f56d4aae000 ---p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aae000-7f56d4aaf000 r--p 000fa000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aaf000-7f56d4ab rw-p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4ab-7f56d4ac6000 r-xp fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4ac6000-7f56d4cc5000 ---p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc5000-7f56d4cc6000 r--p 00015000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc6000-7f56d4cc7000 rw-p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc7000-7f56d4e7c000 r-xp fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d4e7c000-7f56d507c000 ---p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d507c000-7f56d508 r--p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d508-7f56d5082000 rw-p 001b9000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d5082000-7f56d5087000 rw-p 00:00 0 7f56d5087000-7f56d509f000 r-xp fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d509f000-7f56d529e000 ---p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529e000-7f56d529f000 r--p 00017000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529f000-7f56d52a rw-p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d52a-7f56d52a4000 rw-p 00:00 0 7f56d52a4000-7f56d52a6000 r-xp fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d52a6000-7f56d54a6000 ---p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a6000-7f56d54a7000 r--p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a7000-7f56d54a8000 rw-p 3000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a8000-7f56d54db000 r-xp fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d54db000-7f56d56da000 ---p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56da000-7f56d56db000 r--p 00032000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56db000-7f56d56dd000 rw-p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56dd000-7f56d5711000 r-xp fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5711000-7f56d5911000 ---p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5911000-7f56d5912000 r--p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5912000-7f56d5913000 rw-p 00035000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5913000-7f56d5914000 rw-p 00:00 0 7f56d5914000-7f56d5bd2000 r-xp fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5bd2000-7f56d5dd1000 ---p 002be000 fd:01 130807 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 7f56d5dd1000-7f56d5dd7000 r--p 002bd000 fd:01 130807
[pmacct-discussion] nfacctd Networks Problem
Just trying to setup nfacctd to aggregate our traffic on a per host basis. To import into our billing application. Nfacct.conf debug: true ! daemonize: false nfacctd_time_new: true plugins: mysql aggregate: sum_host sql_db: pmacct sql_table: acct sql_table_version: 1 sql_passwd: ** sql_user: ** sql_host: 10.0.8.36 sql_refresh_time: 90 ! sql_optimize_clauses: true sql_history: 10m sql_history_roundoff: mh nfacctd_ip: 10.0.8.40 nfacctd_port: 9996 !logfile: /var/log/nfacctd.log ! sql_preprocess: qnum=1000, minp=5 networks_file: /etc/nfacctd.networks ! ports_file: ./ports.example /etc/nfacctd.networks 192.168.88.0/21 (Not Real Networks) 192.168.40.0/22 Debug Output: [root@pmacct sbin]# nfacctd -f /etc/nfacctd.conf INFO ( default/mysql ): 110592 bytes are available to address shared memory segment; buffer size is 168 bytes. INFO ( default/mysql ): Trying to allocate a shared memory segment of 4644864 bytes. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. INFO ( default/core ): waiting for NetFlow data on 10.0.8.40:9996 ( default/mysql ) *** Purging queries queue *** ( default/mysql ) *** Purging cache - START *** ( default/mysql ) *** Purging cache - END (QN: 0, ET: 0) *** OK: Exiting ... However, when this writes to MySQL it includes all hosts from all networks. The documentation seems straight forward, but it is not working for me. I have also tried limiting to a single /24 network, but still get all hosts. What am I doing wrong? Thanks in Advance; Terry ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Off by one warning?
Hi Joel, Disabling checks is harmless, apart from having the benefit of removing you the annoying part of those warning messages. But one more question: you say sending two streams but i see only a single exporter, 'agent=X:0'. Is it X reallt corresponding to a single IP address (which would justify the warnings) or not? If yes, would it be possible for you to send me privately a brief trace of the export packets from that agent? Cheers, Paolo On Wed, Dec 04, 2013 at 03:53:04PM -0800, Joel Krauska wrote: Paolo, Sorry I missed that you had replied. Yes, these happen all the time. There's a big burst on startup and then a pretty steady one afterwards. It looks like the later burst might be due to sending two streams? INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 6, ET: 0) *** WARN: expecting flow '25593234' but received '234608' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234609' but received '25593234' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593299' but received '234609' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234610' but received '25593299' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593367' but received '234610' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234611' but received '25593367' collector=0.0.0.0:6001agent=X:0 INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 7, ET: 0) *** WARN: expecting flow '25593429' but received '234611' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234612' but received '25593429' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593510' but received '234612' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234613' but received '25593510' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593572' but received '234613' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234614' but received '25593572' collector=0.0.0.0:6001agent=X:0 See how the flow numbers flip back and forth between 234k and 25M? I'm willing to disable checks, but I wouldn't want to miss other debug information in my testing. Cheers, Joel On Mon, Nov 11, 2013 at 3:16 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Joel, Could also be packets are received out of order, which can be harmless depending on the use-cases. Anyway if annoying these messages can be disabled by setting nfacctd_disable_checks to true. I propose this idea because i don't seem to have seen such warnings on a regular basis on other IPFIX exports. Maybe would help if you can define better frequently. Is that like in always, at times, in specific times of the day, or ..? Cheers, Paolo On Sun, Nov 10, 2013 at 06:26:22PM -0800, Joel Krauska wrote: (I should have mentioned I'm testing rc1 NetFlow Accounting Daemon, nfacctd 1.5.0rc1 (20130829-00) --enable-mysql --enable-64bit --enable-threads --enable-geoip I frequently get these Warnings. WARN: expecting flow '4423369' but received '4423371' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423371' but received '4423372' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423372' but received '4423374' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423374' but received '4423375' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423375' but received '4423376' collector=0.0.0.0:6001agent=BLAH:0 It seems odd to see them in series like this, since the 'expected' usually is the one it just received just before... Looks like possibly an off by 1 error? Cheers, Joel ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] buffer overflow / backtrace on 1.5rc1
Hi Joel, Wow, interesting. What OS are you running? What BGP capabilities are enabled and which address families are you sending over? It would help if you can run the daemon under gdb and collect 'bt' information (send it directly to me). Post in the same email also your config. We can take it from there. Cheers, Paolo On Wed, Dec 04, 2013 at 04:18:42PM -0800, Joel Krauska wrote: I get a pretty repeatable buffer overflow when trying to use nfacctd with BGP enabled. (threaded) It will run for a few moments and then bombs out. *** buffer overflow detected ***: nfacctd: Core Process [default] terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f56d4dd1f47] /lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f56d4dd0e40] nfacctd: Core Process [default](bgp_nlri_parse+0x15f)[0x46b17f] nfacctd: Core Process [default](bgp_update_msg+0x3a7)[0x46bb87] nfacctd: Core Process [default](skinny_bgp_daemon+0xc1f)[0x46e84f] nfacctd: Core Process [default](thread_runner+0x5b)[0x45f72b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f56d508ee9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f56d4dbb3fd] === Memory map: 0040-004ca000 r-xp fd:01 270072 /opt/pmacct/sbin/nfacctd 006c9000-006ca000 r--p 000c9000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006ca000-006cb000 rw-p 000ca000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006cb000-0074c000 rw-p 00:00 0 00ab9000-00ada000 rw-p 00:00 0 [heap] 7f56c400-7f56c6e8c000 rw-p 00:00 0 7f56c6e8c000-7f56c800 ---p 00:00 0 7f56cbde2000-7f56cbdf7000 r-xp fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbdf7000-7f56cbff6000 ---p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff6000-7f56cbff7000 r--p 00014000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff7000-7f56cbff8000 rw-p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbfff000-7f56cc00 rw-p 00:00 0 7f56cc00-7f56d000 rw-p 00:00 0 7f56d3a0-7f56d3fb3000 rw-s 00:04 10398639 /dev/zero (deleted) 7f56d3fb3000-7f56d3fb4000 ---p 00:00 0 7f56d3fb4000-7f56d47b4000 rw-p 00:00 0 7f56d47b4000-7f56d48af000 r-xp fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d48af000-7f56d4aae000 ---p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aae000-7f56d4aaf000 r--p 000fa000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aaf000-7f56d4ab rw-p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4ab-7f56d4ac6000 r-xp fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4ac6000-7f56d4cc5000 ---p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc5000-7f56d4cc6000 r--p 00015000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc6000-7f56d4cc7000 rw-p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc7000-7f56d4e7c000 r-xp fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d4e7c000-7f56d507c000 ---p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d507c000-7f56d508 r--p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d508-7f56d5082000 rw-p 001b9000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d5082000-7f56d5087000 rw-p 00:00 0 7f56d5087000-7f56d509f000 r-xp fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d509f000-7f56d529e000 ---p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529e000-7f56d529f000 r--p 00017000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529f000-7f56d52a rw-p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d52a-7f56d52a4000 rw-p 00:00 0 7f56d52a4000-7f56d52a6000 r-xp fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d52a6000-7f56d54a6000 ---p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a6000-7f56d54a7000 r--p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a7000-7f56d54a8000 rw-p 3000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a8000-7f56d54db000 r-xp fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d54db000-7f56d56da000 ---p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56da000-7f56d56db000 r--p 00032000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56db000-7f56d56dd000 rw-p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56dd000-7f56d5711000 r-xp fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5711000-7f56d5911000 ---p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5911000-7f56d5912000 r--p 00034000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5912000-7f56d5913000 rw-p 00035000 fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5913000-7f56d5914000 rw-p 00:00 0 7f56d5914000-7f56d5bd2000 r-xp fd:01
Re: [pmacct-discussion] Off by one warning?
Sure thing. On Wed, Dec 4, 2013 at 4:37 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Joel, Disabling checks is harmless, apart from having the benefit of removing you the annoying part of those warning messages. But one more question: you say sending two streams but i see only a single exporter, 'agent=X:0'. Is it X reallt corresponding to a single IP address (which would justify the warnings) or not? If yes, would it be possible for you to send me privately a brief trace of the export packets from that agent? Cheers, Paolo On Wed, Dec 04, 2013 at 03:53:04PM -0800, Joel Krauska wrote: Paolo, Sorry I missed that you had replied. Yes, these happen all the time. There's a big burst on startup and then a pretty steady one afterwards. It looks like the later burst might be due to sending two streams? INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 6, ET: 0) *** WARN: expecting flow '25593234' but received '234608' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234609' but received '25593234' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593299' but received '234609' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234610' but received '25593299' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593367' but received '234610' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234611' but received '25593367' collector=0.0.0.0:6001agent=X:0 INFO ( testing/print ): *** Purging cache - START *** INFO ( testing/print ): *** Purging cache - END (QN: 7, ET: 0) *** WARN: expecting flow '25593429' but received '234611' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234612' but received '25593429' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593510' but received '234612' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234613' but received '25593510' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '25593572' but received '234613' collector=0.0.0.0:6001agent=X:0 WARN: expecting flow '234614' but received '25593572' collector=0.0.0.0:6001agent=X:0 See how the flow numbers flip back and forth between 234k and 25M? I'm willing to disable checks, but I wouldn't want to miss other debug information in my testing. Cheers, Joel On Mon, Nov 11, 2013 at 3:16 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Joel, Could also be packets are received out of order, which can be harmless depending on the use-cases. Anyway if annoying these messages can be disabled by setting nfacctd_disable_checks to true. I propose this idea because i don't seem to have seen such warnings on a regular basis on other IPFIX exports. Maybe would help if you can define better frequently. Is that like in always, at times, in specific times of the day, or ..? Cheers, Paolo On Sun, Nov 10, 2013 at 06:26:22PM -0800, Joel Krauska wrote: (I should have mentioned I'm testing rc1 NetFlow Accounting Daemon, nfacctd 1.5.0rc1 (20130829-00) --enable-mysql --enable-64bit --enable-threads --enable-geoip I frequently get these Warnings. WARN: expecting flow '4423369' but received '4423371' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423371' but received '4423372' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423372' but received '4423374' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423374' but received '4423375' collector=0.0.0.0:6001agent=BLAH:0 WARN: expecting flow '4423375' but received '4423376' collector=0.0.0.0:6001agent=BLAH:0 It seems odd to see them in series like this, since the 'expected' usually is the one it just received just before... Looks like possibly an off by 1 error? Cheers, Joel ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd Networks Problem
Hi Terry, What version of pmacct are you running? If a recent one, ie. = 0.14.3, you should have 'networks_file_filter: true' in your config in order to explicitely enable filtering (as is documented in the CONFIG-KEYS file). Cheers, Paolo On Thu, Dec 05, 2013 at 12:37:58AM +, Terry Duchcherer wrote: Just trying to setup nfacctd to aggregate our traffic on a per host basis. To import into our billing application. Nfacct.conf debug: true ! daemonize: false nfacctd_time_new: true plugins: mysql aggregate: sum_host sql_db: pmacct sql_table: acct sql_table_version: 1 sql_passwd: ** sql_user: ** sql_host: 10.0.8.36 sql_refresh_time: 90 ! sql_optimize_clauses: true sql_history: 10m sql_history_roundoff: mh nfacctd_ip: 10.0.8.40 nfacctd_port: 9996 !logfile: /var/log/nfacctd.log ! sql_preprocess: qnum=1000, minp=5 networks_file: /etc/nfacctd.networks ! ports_file: ./ports.example /etc/nfacctd.networks 192.168.88.0/21 (Not Real Networks) 192.168.40.0/22 Debug Output: [root@pmacct sbin]# nfacctd -f /etc/nfacctd.conf INFO ( default/mysql ): 110592 bytes are available to address shared memory segment; buffer size is 168 bytes. INFO ( default/mysql ): Trying to allocate a shared memory segment of 4644864 bytes. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. INFO ( default/core ): waiting for NetFlow data on 10.0.8.40:9996 ( default/mysql ) *** Purging queries queue *** ( default/mysql ) *** Purging cache - START *** ( default/mysql ) *** Purging cache - END (QN: 0, ET: 0) *** OK: Exiting ... However, when this writes to MySQL it includes all hosts from all networks. The documentation seems straight forward, but it is not working for me. I have also tried limiting to a single /24 network, but still get all hosts. What am I doing wrong? Thanks in Advance; Terry ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] buffer overflow / backtrace on 1.5rc1
Running Ubuntu 12.04.2. Can't speak to the BGP config on the Juniper side, but I'll ask for that. I believe IPv6 might be in the mix here. I'll do the gdb bt and email you. On Wed, Dec 4, 2013 at 4:46 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Joel, Wow, interesting. What OS are you running? What BGP capabilities are enabled and which address families are you sending over? It would help if you can run the daemon under gdb and collect 'bt' information (send it directly to me). Post in the same email also your config. We can take it from there. Cheers, Paolo On Wed, Dec 04, 2013 at 04:18:42PM -0800, Joel Krauska wrote: I get a pretty repeatable buffer overflow when trying to use nfacctd with BGP enabled. (threaded) It will run for a few moments and then bombs out. *** buffer overflow detected ***: nfacctd: Core Process [default] terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f56d4dd1f47] /lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f56d4dd0e40] nfacctd: Core Process [default](bgp_nlri_parse+0x15f)[0x46b17f] nfacctd: Core Process [default](bgp_update_msg+0x3a7)[0x46bb87] nfacctd: Core Process [default](skinny_bgp_daemon+0xc1f)[0x46e84f] nfacctd: Core Process [default](thread_runner+0x5b)[0x45f72b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f56d508ee9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f56d4dbb3fd] === Memory map: 0040-004ca000 r-xp fd:01 270072 /opt/pmacct/sbin/nfacctd 006c9000-006ca000 r--p 000c9000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006ca000-006cb000 rw-p 000ca000 fd:01 270072 /opt/pmacct/sbin/nfacctd 006cb000-0074c000 rw-p 00:00 0 00ab9000-00ada000 rw-p 00:00 0 [heap] 7f56c400-7f56c6e8c000 rw-p 00:00 0 7f56c6e8c000-7f56c800 ---p 00:00 0 7f56cbde2000-7f56cbdf7000 r-xp fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbdf7000-7f56cbff6000 ---p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff6000-7f56cbff7000 r--p 00014000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbff7000-7f56cbff8000 rw-p 00015000 fd:01 390697 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f56cbfff000-7f56cc00 rw-p 00:00 0 7f56cc00-7f56d000 rw-p 00:00 0 7f56d3a0-7f56d3fb3000 rw-s 00:04 10398639 /dev/zero (deleted) 7f56d3fb3000-7f56d3fb4000 ---p 00:00 0 7f56d3fb4000-7f56d47b4000 rw-p 00:00 0 7f56d47b4000-7f56d48af000 r-xp fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d48af000-7f56d4aae000 ---p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aae000-7f56d4aaf000 r--p 000fa000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4aaf000-7f56d4ab rw-p 000fb000 fd:01 394152 /lib/x86_64-linux-gnu/libm-2.15.so 7f56d4ab-7f56d4ac6000 r-xp fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4ac6000-7f56d4cc5000 ---p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc5000-7f56d4cc6000 r--p 00015000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc6000-7f56d4cc7000 rw-p 00016000 fd:01 390743 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f56d4cc7000-7f56d4e7c000 r-xp fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d4e7c000-7f56d507c000 ---p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d507c000-7f56d508 r--p 001b5000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d508-7f56d5082000 rw-p 001b9000 fd:01 394141 /lib/x86_64-linux-gnu/libc-2.15.so 7f56d5082000-7f56d5087000 rw-p 00:00 0 7f56d5087000-7f56d509f000 r-xp fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d509f000-7f56d529e000 ---p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529e000-7f56d529f000 r--p 00017000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d529f000-7f56d52a rw-p 00018000 fd:01 394150 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f56d52a-7f56d52a4000 rw-p 00:00 0 7f56d52a4000-7f56d52a6000 r-xp fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d52a6000-7f56d54a6000 ---p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a6000-7f56d54a7000 r--p 2000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a7000-7f56d54a8000 rw-p 3000 fd:01 394156 /lib/x86_64-linux-gnu/libdl-2.15.so 7f56d54a8000-7f56d54db000 r-xp fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d54db000-7f56d56da000 ---p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56da000-7f56d56db000 r--p 00032000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56db000-7f56d56dd000 rw-p 00033000 fd:01 131070 /usr/lib/libGeoIP.so.1.4.8 7f56d56dd000-7f56d5711000 r-xp fd:01 135098 /usr/lib/x86_64-linux-gnu/libpcap.so.1.1.1 7f56d5711000-7f56d5911000 ---p 00034000
Re: [pmacct-discussion] nfacctd Networks Problem
Thank you, that was my problem. Terry -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Paolo Lucente Sent: Wednesday, December 04, 2013 5:53 PM To: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] nfacctd Networks Problem Hi Terry, What version of pmacct are you running? If a recent one, ie. = 0.14.3, you should have 'networks_file_filter: true' in your config in order to explicitely enable filtering (as is documented in the CONFIG-KEYS file). Cheers, Paolo On Thu, Dec 05, 2013 at 12:37:58AM +, Terry Duchcherer wrote: Just trying to setup nfacctd to aggregate our traffic on a per host basis. To import into our billing application. Nfacct.conf debug: true ! daemonize: false nfacctd_time_new: true plugins: mysql aggregate: sum_host sql_db: pmacct sql_table: acct sql_table_version: 1 sql_passwd: ** sql_user: ** sql_host: 10.0.8.36 sql_refresh_time: 90 ! sql_optimize_clauses: true sql_history: 10m sql_history_roundoff: mh nfacctd_ip: 10.0.8.40 nfacctd_port: 9996 !logfile: /var/log/nfacctd.log ! sql_preprocess: qnum=1000, minp=5 networks_file: /etc/nfacctd.networks ! ports_file: ./ports.example /etc/nfacctd.networks 192.168.88.0/21 (Not Real Networks) 192.168.40.0/22 Debug Output: [root@pmacct sbin]# nfacctd -f /etc/nfacctd.conf INFO ( default/mysql ): 110592 bytes are available to address shared memory segment; buffer size is 168 bytes. INFO ( default/mysql ): Trying to allocate a shared memory segment of 4644864 bytes. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.40.0 mask: 22 DEBUG ( /etc/nfacctd.networks ): [networks table IPv4] nh: asn: 0 net: 192.168.88.0 mask: 21 DEBUG ( /etc/nfacctd.networks ): IPv4 Networks Cache successfully created: 1 entries. INFO ( default/core ): waiting for NetFlow data on 10.0.8.40:9996 ( default/mysql ) *** Purging queries queue *** ( default/mysql ) *** Purging cache - START *** ( default/mysql ) *** Purging cache - END (QN: 0, ET: 0) *** OK: Exiting ... However, when this writes to MySQL it includes all hosts from all networks. The documentation seems straight forward, but it is not working for me. I have also tried limiting to a single /24 network, but still get all hosts. What am I doing wrong? Thanks in Advance; Terry ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
