Re: [pmacct-discussion] a single aggregate misses almost all traffic
Hi, For the archives: we found out some traffic was VLAN tagged, hence defeating the aggregate_filter. Johannes to investigate and keep in touch if anything on the pmacctd side of the things is wrong. Cheers, Paolo On Wed, Apr 09, 2014 at 12:01:47AM +0200, Johannes Formann wrote: Hi Paolo, The issue is persistent across reloads, correct? Correct. And since the other aggregates seem to work normal I hope that this time the hardware is not the reason ;) Enable debug only for the plugin, ie. debug[TMPflowSRC]: true, then check the log file to see if the problem is somehow with the database. Here are the results: Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( default/core ): Start logging ... Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( inbound/mysql ): plugin_pipe_size=4096 bytes plugin_buffer_size=163840 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( inbound/mysql ): ctrl channel: obtained=212992 bytes target=2000 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( outbound/mysql ): plugin_pipe_size=4096 bytes plugin_buffer_size=163840 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( outbound/mysql ): ctrl channel: obtained=212992 bytes target=2000 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( TMPflowSRC/mysql ): plugin_pipe_size=4096 bytes plugin_buffer_size=163840 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( TMPflowSRC/mysql ): ctrl channel: obtained=212992 bytes target=2000 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( TMPflowDST/mysql ): plugin_pipe_size=4096 bytes plugin_buffer_size=163840 bytes Apr 8 21:25:17 traffic pmacctd[15489]: INFO ( TMPflowDST/mysql ): ctrl channel: obtained=212992 bytes target=2000 bytes Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.64.0 mask: 20 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.48.0 mask: 20 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.164.0 mask: 24 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.4.0 mask: 22 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.104.0 mask: 22 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: xxx.yyy.114.0 mask: 23 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): IPv4 Networks Cache successfully created: 1 entries. Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): [networks table IPv6] nh: peer_asn: 0 asn: 0 net: 2a00:xyza:: mask: 32 Apr 8 21:25:17 traffic pmacctd[15492]: DEBUG ( /etc/pmacct/networks ): IPv6 Networks Cache successfully created: 32771 entries. Apr 8 21:25:17 traffic kernel: [22097.047614] device eth0 entered promiscuous mode Apr 8 21:25:17 traffic pmacctd[15489]: OK ( default/core ): link type is: 1 Apr 8 21:26:01 traffic pmacctd[15494]: INFO ( TMPflowDST/mysql ): *** Purging cache - START (PID: 15494) *** Apr 8 21:26:01 traffic pmacctd[15495]: INFO ( TMPflowSRC/mysql ): *** Purging cache - START (PID: 15495) *** Apr 8 21:26:01 traffic pmacctd[15499]: INFO ( outbound/mysql ): *** Purging cache - START (PID: 15499) *** Apr 8 21:26:01 traffic pmacctd[15498]: INFO ( inbound/mysql ): *** Purging cache - START (PID: 15498) *** Apr 8 21:26:01 traffic pmacctd[15495]: DEBUG ( TMPflowSRC/mysql ): 36 VALUES statements sent to the MySQL server. Apr 8 21:26:01 traffic pmacctd[15495]: INFO ( TMPflowSRC/mysql ): *** Purging cache - END (PID: 15495, QN: 36, ET: 0) *** Apr 8 21:26:01 traffic pmacctd[15494]: INFO ( TMPflowDST/mysql ): *** Purging cache - END (PID: 15494, QN: 556, ET: 0) *** Apr 8 21:26:01 traffic pmacctd[15498]: INFO ( inbound/mysql ): *** Purging cache - END (PID: 15498, QN: 1878, ET: 0) *** Apr 8 21:26:01 traffic pmacctd[15499]: INFO ( outbound/mysql ): *** Purging cache - END (PID: 15499, QN: 2333, ET: 0) *** Apr 8 21:27:01 traffic pmacctd[15520]: INFO ( outbound/mysql ): *** Purging cache - START (PID: 15520) *** Apr 8 21:27:01 traffic pmacctd[15521]: INFO ( inbound/mysql ): *** Purging cache - START (PID: 15521) *** Apr 8 21:27:01 traffic pmacctd[15524]: INFO ( TMPflowSRC/mysql ): *** Purging cache - START (PID: 15524) *** Apr 8 21:27:01 traffic pmacctd[15526]: INFO ( TMPflowDST/mysql ): *** Purging cache - START (PID: 15526) *** Apr 8 21:27:01 traffic pmacctd[15524]: DEBUG ( TMPflowSRC/mysql ): 57 VALUES statements sent to the MySQL server. Apr 8 21:27:01 traffic pmacctd[15524]: INFO ( TMPflowSRC/mysql ): *** Purging cache - END (PID: 15524, QN: 57, ET: 0) *** Apr 8 21:27:01
[pmacct-discussion] a single aggregate misses almost all traffic
Hi, I have a strange problem again. I already tested the newest CVS version but it persists: I use four aggregates: - inbound: incoming traffic for local IPs - outbound: outgoing traffic for local ips - TMPflowSRC: short time local outgoing udp traffic (with a short port list) - TMPflowDST: short time udp traffic (after destination address) to be able to identify potential outgoing ddos Everything worked fine for some time but today I found that TMPflowSRC accounts almost no traffic anymore and if its accounts mostly IPv6 addresses. Sadly in this network IPv6 traffic is less than 4% so there must be some fault in my configuration. With debug enabled the initialization of the three aggregates with a network filter looks identical, so there shouldn’t be the problem. commenting out the port-List didn’t help either. My configuration: ! pmacctd configuration ! ! ! daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon ! aggregate[inbound]: dst_host aggregate[outbound]: src_host,proto aggregate[TMPflowSRC]: src_host,src_port,proto aggregate[TMPflowDST]: dst_host,proto aggregate_filter[TMPflowSRC]: udp aggregate_filter[TMPflowDST]: udp plugins: mysql[inbound], mysql[outbound], mysql[TMPflowSRC], mysql[TMPflowDST] sql_table[inbound]: acct_%Y_%m_in sql_table[outbound]: acct_%Y_%m_out sql_table[TMPflowSRC]: acct_TMPflowSRC sql_table[TMPflowDST]: acct_TMPflowDST sql_table_schema[inbound]: /etc/pmacct/inbound.schema sql_table_schema[outbound]: /etc/pmacct/outbound.schema networks_file[inbound]: /etc/pmacct/networks networks_file[outbound]: /etc/pmacct/networks networks_file[TMPflowSRC]: /etc/pmacct/networks ports_file[TMPflowSRC]: /etc/pmacct/portsudp networks_file_filter: true interface: eth0 ! storage methods sql_db: pmacct sql_table_version: 4 sql_passwd: secret sql_user: pmacct sql_refresh_time: 60 sql_optimize_clauses: true sql_history: 1m sql_history_roundoff: m sql_multi_values: 1200 sql_cache_entries: 64000 pmacctd_flow_buffer_buckets: 4096 sql_dont_try_update: true plugin_buffer_size: 163840 plugin_pipe_size: 4096 Any ideas where to look? greetings Johannes ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] a single aggregate misses almost all traffic
Hi Johannes, The issue is persistent across reloads, correct? Enable debug only for the plugin, ie. debug[TMPflowSRC]: true, then check the log file to see if the problem is somehow with the database. If log is clear, meaning no entries are written out, i'd recommend to remove anything that can be potentially filtering so: aggregate_filter, networks_file and ports_file: see if this changes anything. Let's take it from there. Cheers, Paolo On Tue, Apr 08, 2014 at 05:50:50PM +0200, Johannes Formann wrote: Hi, I have a strange problem again. I already tested the newest CVS version but it persists: I use four aggregates: - inbound: incoming traffic for local IPs - outbound: outgoing traffic for local ips - TMPflowSRC: short time local outgoing udp traffic (with a short port list) - TMPflowDST: short time udp traffic (after destination address) to be able to identify potential outgoing ddos Everything worked fine for some time but today I found that TMPflowSRC accounts almost no traffic anymore and if its accounts mostly IPv6 addresses. Sadly in this network IPv6 traffic is less than 4% so there must be some fault in my configuration. With debug enabled the initialization of the three aggregates with a network filter looks identical, so there shouldn’t be the problem. commenting out the port-List didn’t help either. My configuration: ! pmacctd configuration ! ! ! daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon ! aggregate[inbound]: dst_host aggregate[outbound]: src_host,proto aggregate[TMPflowSRC]: src_host,src_port,proto aggregate[TMPflowDST]: dst_host,proto aggregate_filter[TMPflowSRC]: udp aggregate_filter[TMPflowDST]: udp plugins: mysql[inbound], mysql[outbound], mysql[TMPflowSRC], mysql[TMPflowDST] sql_table[inbound]: acct_%Y_%m_in sql_table[outbound]: acct_%Y_%m_out sql_table[TMPflowSRC]: acct_TMPflowSRC sql_table[TMPflowDST]: acct_TMPflowDST sql_table_schema[inbound]: /etc/pmacct/inbound.schema sql_table_schema[outbound]: /etc/pmacct/outbound.schema networks_file[inbound]: /etc/pmacct/networks networks_file[outbound]: /etc/pmacct/networks networks_file[TMPflowSRC]: /etc/pmacct/networks ports_file[TMPflowSRC]: /etc/pmacct/portsudp networks_file_filter: true interface: eth0 ! storage methods sql_db: pmacct sql_table_version: 4 sql_passwd: secret sql_user: pmacct sql_refresh_time: 60 sql_optimize_clauses: true sql_history: 1m sql_history_roundoff: m sql_multi_values: 1200 sql_cache_entries: 64000 pmacctd_flow_buffer_buckets: 4096 sql_dont_try_update: true plugin_buffer_size: 163840 plugin_pipe_size: 4096 Any ideas where to look? greetings Johannes ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
