Re: [pmacct-discussion] Trouble with average packetsize
Hi all, after a lot of testing and changing everything (pmacct version, newer libpcap, kernel, Motherboard, switch) the reason behind the problem was quite trivial: The network card (intel 1000e series) has some hardware acceleration functions that can assemble fragmented packets bevor showing them to the operating system… The following commands disable these functions, and after that the results are reasonable again. /sbin/ethtool -K eth0 tso off /sbin/ethtool -K eth0 gso off /sbin/ethtool -K eth0 gro off cheers Johannes Am 26.02.2014 um 20:25 schrieb Johannes Formann mlm...@formann.de: Hi Paolo, Thanks for your response. I tried the SVS-Version (is the repository on the website the right one, it has 0.14 in the path) and have the same problems. Changing the buffer to the size you suggested don’t help too. Build it using this commands: mkdir pmacct-latest cd pmacct-latest/ CVSROOT=:pserver:anonym...@cvs.pmacct.net:2402/home/repo-0.14 export CVSROOT cvs login cvs co pmacct cd pmacct/ ./configure --enable-l2=no --enable-ipv6 --enable-mysql make make install changed the /etc/init.d/pmacct the DAEMON-Line to use /usr/local/sbin/pmacctd service pmacct restart I stumpled over an other bug too, the networks-File is ignored. I think remote access should be possible but I have to ask someone else before giving you login credentials. cheers Johannes Am 26.02.2014 um 19:04 schrieb Paolo Lucente pa...@pmacct.net: Hi Johannes, This is indeed strange. I have two leads: there was a bug around syslog logging being fixed recently (post 1.5.0rc2); the bug was causing some memory corruption: can you, for a try, log directly to a file via 'logfile' directive (or alternatively, slightly more involved, upgrade to latesst CVS build). Should that not do can you please try introducing some minimal internal buffering? Something like the following should do: plugin_pipe_size: 4096000 plugin_buffer_size: 40960 Let me know. Should none of the above work, would you be open to some remote-access for a bit of debugging? Cheers, Paolo On Tue, Feb 25, 2014 at 11:46:31PM +0100, Johannes Formann wrote: Hello, I try to run pmacct on a debian 7 box. Basically everything runs fine, except sometimes pmacctd mud miscount the traffic, since the average packetize goes over 1500 (latest size observed about 7.000) wich is not possible (MTU-Limit of 1500 bytes). Same problem with the debian packages an a fresh compiled version from 1.5.rc2. Any Ideas? regards Johannes P.S. Used config ! pmacctd configuration ! ! daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon aggregate[inbound]: dst_host aggregate[outbound]: src_host,proto aggregate[TMPflowSRC]: src_host,src_port,proto aggregate[TMPflowDST]: dst_host,proto aggregate_filter[TMPflowSRC]: udp aggregate_filter[TMPflowDST]: udp plugins: mysql[inbound], mysql[outbound], mysql[TMPflowSRC], mysql[TMPflowDST] sql_table[inbound]: acct_%Y_%m_in sql_table[outbound]: acct_%Y_%m_out sql_table[TMPflowSRC]: acct_TMPflowSRC sql_table[TMPflowDST]: acct_TMPflowDST sql_table_schema[inbound]: /etc/pmacct/inbound.schema sql_table_schema[outbound]: /etc/pmacct/outbound.schema networks_file[inbound]: /etc/pmacct/networks networks_file[outbound]: /etc/pmacct/networks networks_file[TMPflowSRC]: /etc/pmacct/networks ports_file[TMPflowSRC]: /etc/pmacct/ports-udp ! on this interface interface: eth0 sql_db: pmacct sql_table_version: 4 sql_passwd: secret ;) sql_user: pmacct ! refresh the db every 1 minute sql_refresh_time: 60 ! reduce the size of the insert/update clause sql_optimize_clauses: true ! accumulate values in each row for up to 5 minutes sql_history: 1m ! create new rows on the minute, hour, day boundaries sql_history_roundoff: m ! Multivalue-Support, bei mysql schneller sql_multi_values: 1200 ! viele Daten hier, mehr puffer sql_cache_entries: 64000 ports_file: /etc/pmacct/ports ! nur inserts sql_dont_try_update: true ! intern puffern plugin_buffer_size: 163840 plugin_pipe_size: 4096 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Trouble with average packetsize
Hi Johannes, This is indeed strange. I have two leads: there was a bug around syslog logging being fixed recently (post 1.5.0rc2); the bug was causing some memory corruption: can you, for a try, log directly to a file via 'logfile' directive (or alternatively, slightly more involved, upgrade to latesst CVS build). Should that not do can you please try introducing some minimal internal buffering? Something like the following should do: plugin_pipe_size: 4096000 plugin_buffer_size: 40960 Let me know. Should none of the above work, would you be open to some remote-access for a bit of debugging? Cheers, Paolo On Tue, Feb 25, 2014 at 11:46:31PM +0100, Johannes Formann wrote: Hello, I try to run pmacct on a debian 7 box. Basically everything runs fine, except sometimes pmacctd mud miscount the traffic, since the average packetize goes over 1500 (latest size observed about 7.000) wich is not possible (MTU-Limit of 1500 bytes). Same problem with the debian packages an a fresh compiled version from 1.5.rc2. Any Ideas? regards Johannes P.S. Used config ! pmacctd configuration ! ! daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon aggregate[inbound]: dst_host aggregate[outbound]: src_host,proto aggregate[TMPflowSRC]: src_host,src_port,proto aggregate[TMPflowDST]: dst_host,proto aggregate_filter[TMPflowSRC]: udp aggregate_filter[TMPflowDST]: udp plugins: mysql[inbound], mysql[outbound], mysql[TMPflowSRC], mysql[TMPflowDST] sql_table[inbound]: acct_%Y_%m_in sql_table[outbound]: acct_%Y_%m_out sql_table[TMPflowSRC]: acct_TMPflowSRC sql_table[TMPflowDST]: acct_TMPflowDST sql_table_schema[inbound]: /etc/pmacct/inbound.schema sql_table_schema[outbound]: /etc/pmacct/outbound.schema networks_file[inbound]: /etc/pmacct/networks networks_file[outbound]: /etc/pmacct/networks networks_file[TMPflowSRC]: /etc/pmacct/networks ports_file[TMPflowSRC]: /etc/pmacct/ports-udp ! on this interface interface: eth0 sql_db: pmacct sql_table_version: 4 sql_passwd: secret ;) sql_user: pmacct ! refresh the db every 1 minute sql_refresh_time: 60 ! reduce the size of the insert/update clause sql_optimize_clauses: true ! accumulate values in each row for up to 5 minutes sql_history: 1m ! create new rows on the minute, hour, day boundaries sql_history_roundoff: m ! Multivalue-Support, bei mysql schneller sql_multi_values: 1200 ! viele Daten hier, mehr puffer sql_cache_entries: 64000 ports_file: /etc/pmacct/ports ! nur inserts sql_dont_try_update: true ! intern puffern plugin_buffer_size: 163840 plugin_pipe_size: 4096 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Trouble with average packetsize
Hi Paolo, Thanks for your response. I tried the SVS-Version (is the repository on the website the right one, it has 0.14 in the path) and have the same problems. Changing the buffer to the size you suggested don’t help too. Build it using this commands: mkdir pmacct-latest cd pmacct-latest/ CVSROOT=:pserver:anonym...@cvs.pmacct.net:2402/home/repo-0.14 export CVSROOT cvs login cvs co pmacct cd pmacct/ ./configure --enable-l2=no --enable-ipv6 --enable-mysql make make install changed the /etc/init.d/pmacct the DAEMON-Line to use /usr/local/sbin/pmacctd service pmacct restart I stumpled over an other bug too, the networks-File is ignored. I think remote access should be possible but I have to ask someone else before giving you login credentials. cheers Johannes Am 26.02.2014 um 19:04 schrieb Paolo Lucente pa...@pmacct.net: Hi Johannes, This is indeed strange. I have two leads: there was a bug around syslog logging being fixed recently (post 1.5.0rc2); the bug was causing some memory corruption: can you, for a try, log directly to a file via 'logfile' directive (or alternatively, slightly more involved, upgrade to latesst CVS build). Should that not do can you please try introducing some minimal internal buffering? Something like the following should do: plugin_pipe_size: 4096000 plugin_buffer_size: 40960 Let me know. Should none of the above work, would you be open to some remote-access for a bit of debugging? Cheers, Paolo On Tue, Feb 25, 2014 at 11:46:31PM +0100, Johannes Formann wrote: Hello, I try to run pmacct on a debian 7 box. Basically everything runs fine, except sometimes pmacctd mud miscount the traffic, since the average packetize goes over 1500 (latest size observed about 7.000) wich is not possible (MTU-Limit of 1500 bytes). Same problem with the debian packages an a fresh compiled version from 1.5.rc2. Any Ideas? regards Johannes P.S. Used config ! pmacctd configuration ! ! daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon aggregate[inbound]: dst_host aggregate[outbound]: src_host,proto aggregate[TMPflowSRC]: src_host,src_port,proto aggregate[TMPflowDST]: dst_host,proto aggregate_filter[TMPflowSRC]: udp aggregate_filter[TMPflowDST]: udp plugins: mysql[inbound], mysql[outbound], mysql[TMPflowSRC], mysql[TMPflowDST] sql_table[inbound]: acct_%Y_%m_in sql_table[outbound]: acct_%Y_%m_out sql_table[TMPflowSRC]: acct_TMPflowSRC sql_table[TMPflowDST]: acct_TMPflowDST sql_table_schema[inbound]: /etc/pmacct/inbound.schema sql_table_schema[outbound]: /etc/pmacct/outbound.schema networks_file[inbound]: /etc/pmacct/networks networks_file[outbound]: /etc/pmacct/networks networks_file[TMPflowSRC]: /etc/pmacct/networks ports_file[TMPflowSRC]: /etc/pmacct/ports-udp ! on this interface interface: eth0 sql_db: pmacct sql_table_version: 4 sql_passwd: secret ;) sql_user: pmacct ! refresh the db every 1 minute sql_refresh_time: 60 ! reduce the size of the insert/update clause sql_optimize_clauses: true ! accumulate values in each row for up to 5 minutes sql_history: 1m ! create new rows on the minute, hour, day boundaries sql_history_roundoff: m ! Multivalue-Support, bei mysql schneller sql_multi_values: 1200 ! viele Daten hier, mehr puffer sql_cache_entries: 64000 ports_file: /etc/pmacct/ports ! nur inserts sql_dont_try_update: true ! intern puffern plugin_buffer_size: 163840 plugin_pipe_size: 4096 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
