Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Chris, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Your specific configuration is tricky because you write to the RDBMS every 60 seconds but aggregate on 5 minutes time-bins and hence it's not possible to determine for sure whether a record is contributing or not to a certain aggregate. Only work-around i can think of for your case is to permanenly enable debug in pmacct (or logging in the RDBMS) so that it's possible, at any moment, to know which queries have been performed. This at the expense of slightly more resources. If using default SQL schemas and an INSERT-only scenario, to protect against duplicates, you could have simply appended the '-i' option to pmpgplay to disable UPDATE SQL queries. Logics is explained here: http://wiki.pmacct.net/CustomizingTheSqlIndexes Cheers, Paolo On Fri, Jul 23, 2010 at 11:53:53AM +1000, Chris wrote: Hi, I'm using pmacct-0.12.0rc3 with postgresql. I've been trying to work out what pmpgplay does when it looks at a log file. I couldn't find much in my searches that go into the details of how it does things. It seems it generates a bunch of sql, but does it check to see if that record exists already before applying the change? I'm worried about replaying the log and it double-recording data (ie the data is already recorded in the db, and then replaying the log does it again), or will it only apply new records that it can't find in the db? I have a missing chunk of data from my database (about 1.5 hrs worth) that I can see in the logs, but the log is surrounded by data that could be in the database (I haven't checked each individual log entry). Relevant pmacct.conf info: plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 sql_table_version:1 sql_refresh_time: 60 sql_history: 5m Any information is much appreciated, thanks! -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [nfacctd] IPv4/IPv6 Accounting into MySQL
Hi Carsten, A couple of things come to mind: * Was pmacct compiled with --enable-ipv6 ? * Are 'aggregate_filter' directives including the IPv6 subnets you want to account for? * If, in parallel to nfacctd, you fire up a Wireshark (or tshark); by decoding the NetFlow v9 datagrams generated by the probe, can you spot the IPv6 records being sent over to the collector? Basing on the answers to the above questions, we can drill it down further. Cheers, Paolo On Fri, Jul 23, 2010 at 07:00:00PM +0200, InterNetX - Carsten Schoene wrote: Hello List, i've setup a nfacctd instance which writes data into mysql, the configuration looks like this: daemonize: true pidfile: /var/run/nfacctd.pid syslog: daemon plugin_pipe_size: 1024 plugin_buffer_size: 10240 nfacctd_ip: %MY_NFACCTD_HOST% nfacctd_port: 5552 nfacctd_allow_file:/etc/pmacct/nfacctd.allow nfacctd_time_new: true plugins: mysql[in], mysql[out] aggregate[in]: dst_host aggregate[out]: src_host aggregate_filter[in]: dst net %MYNETWORK_A%/18 or dst net %MYNETWORK_B%/19 aggregate_filter[out]: src net %MYNETWORK_A%/18 or src net %MYNETWORK_B%/19 sql_host:localhost sql_user:pmacct sql_passwd:* sql_optimize_clauses: true sql_refresh_time: 300 sql_history: 5m sql_dont_try_update: true sql_history_roundoff: m sql_table_version: 7 sql_table:acct_v7_%Y%m%d_%H00 sql_table_schema:/etc/pmacct/mysql_v7_table.sql sql_recovery_logfile: /raid/nfacctd_recovery_log sql_multi_values: 16384000 CREATE TABLE IF NOT EXISTS acct_v7_%Y%m%d_%H00 ( id INT UNSIGNED NOT NULL AUTO_INCREMENT, ip_src CHAR(45) NOT NULL, ip_dst CHAR(45) NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, flows INT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (id), KEY `src_dst_stamps` (ip_src, ip_dst, stamp_inserted, stamp_updated), KEY(ip_src), KEY(ip_dst) ) ENGINE=MyISAM; We are running nprobe which exports netflow v9 data using the following netflow template: %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV4_SRC_ADDR %IPV4_DST_ADDR %LAST_SWITCHED %FIRST_SWITCHED %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %PROTOCOL %TCP_FLAGS %IP_PROTOCOL_VERSION %SRC_TOS %SRC_AS %DST_AS %IPV6_SRC_MASK %IPV6_DST_MASK %SRC_MASK %DST_MASK The problem is, that there are no IPv6 entrys written into the tables, only ipv4 addresses. Any idea or help would be great Regards -- Carsten Sch?ne Leiter Rechenzentrum InterNetX GmbH Maximilianstr. 6 93047 Regensburg Tel. +49 941 59559-480 Fax +49 941 59579-051 www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Gesch?ftsf?hrer/CEO: Thomas M?rz Amtsgericht Regensburg, HRB 7142 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [nfacctd] IPv4/IPv6 Accounting into MySQL
Hello Paolo, thanke for your fast reply, now to your questions: * yes, --enable-ipv6 was used at compile time * i have tried with and without aggregate_filter for our IPv6 subnet * yes, i can see netflow v9 datagrams containing IPv6 records I hope this helps, if you need a sample datagram i can send it over on monday. Regards Carsten Zitat von Paolo Lucente pa...@pmacct.net: Hi Carsten, A couple of things come to mind: * Was pmacct compiled with --enable-ipv6 ? * Are 'aggregate_filter' directives including the IPv6 subnets you want to account for? * If, in parallel to nfacctd, you fire up a Wireshark (or tshark); by decoding the NetFlow v9 datagrams generated by the probe, can you spot the IPv6 records being sent over to the collector? Basing on the answers to the above questions, we can drill it down further. Cheers, Paolo On Fri, Jul 23, 2010 at 07:00:00PM +0200, InterNetX - Carsten Schoene wrote: Hello List, i've setup a nfacctd instance which writes data into mysql, the configuration looks like this: daemonize: true pidfile: /var/run/nfacctd.pid syslog: daemon plugin_pipe_size: 1024 plugin_buffer_size: 10240 nfacctd_ip: %MY_NFACCTD_HOST% nfacctd_port: 5552 nfacctd_allow_file:/etc/pmacct/nfacctd.allow nfacctd_time_new: true plugins: mysql[in], mysql[out] aggregate[in]: dst_host aggregate[out]: src_host aggregate_filter[in]: dst net %MYNETWORK_A%/18 or dst net %MYNETWORK_B%/19 aggregate_filter[out]: src net %MYNETWORK_A%/18 or src net %MYNETWORK_B%/19 sql_host:localhost sql_user:pmacct sql_passwd:* sql_optimize_clauses: true sql_refresh_time: 300 sql_history: 5m sql_dont_try_update: true sql_history_roundoff: m sql_table_version: 7 sql_table:acct_v7_%Y%m%d_%H00 sql_table_schema:/etc/pmacct/mysql_v7_table.sql sql_recovery_logfile: /raid/nfacctd_recovery_log sql_multi_values: 16384000 CREATE TABLE IF NOT EXISTS acct_v7_%Y%m%d_%H00 ( id INT UNSIGNED NOT NULL AUTO_INCREMENT, ip_src CHAR(45) NOT NULL, ip_dst CHAR(45) NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, flows INT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (id), KEY `src_dst_stamps` (ip_src, ip_dst, stamp_inserted, stamp_updated), KEY(ip_src), KEY(ip_dst) ) ENGINE=MyISAM; We are running nprobe which exports netflow v9 data using the following netflow template: %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV4_SRC_ADDR %IPV4_DST_ADDR %LAST_SWITCHED %FIRST_SWITCHED %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %PROTOCOL %TCP_FLAGS %IP_PROTOCOL_VERSION %SRC_TOS %SRC_AS %DST_AS %IPV6_SRC_MASK %IPV6_DST_MASK %SRC_MASK %DST_MASK The problem is, that there are no IPv6 entrys written into the tables, only ipv4 addresses. Any idea or help would be great Regards -- Carsten Sch?ne Leiter Rechenzentrum InterNetX GmbH Maximilianstr. 6 93047 Regensburg Tel. +49 941 59559-480 Fax +49 941 59579-051 www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Gesch?ftsf?hrer/CEO: Thomas M?rz Amtsgericht Regensburg, HRB 7142 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- Carsten Schöne Leiter Rechenzentrum InterNetX GmbH Maximilianstr. 6 93047 Regensburg Tel. +49 941 59559-0 Fax +49 941 59559-50 E-Mail: carsten.scho...@internetx.de www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Geschäftsführer/CEO: Thomas Mörz Amtsgericht Regensburg, HRB 7142 This message was sent using IMP, the Internet Messaging Program. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
