Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Chris, On Mon, Jul 26, 2010 at 10:03:15AM +1000, Chris wrote: So I'd be better off making these 2 match? sql_refresh_time: 60 sql_history: 5m Correct. For what it's worth the log seems to be aggregated on the history time. I only see the 5 min increments when I run pmpgplay in debug mode. Yes, this is because you get logged down the exact tuple that was going to be written to the database. And records in the database are fitted in time bins as per the sql_history parameter. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Paolo, For what it's worth the log seems to be aggregated on the history time. I only see the 5 min increments when I run pmpgplay in debug mode. Yes, this is because you get logged down the exact tuple that was going to be written to the database. And records in the database are fitted in time bins as per the sql_history parameter. Thanks for all the info - very helpful :D -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmpgplay - what does it do?
Hi, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Excellent, thanks. Your specific configuration is tricky because you write to the RDBMS every 60 seconds but aggregate on 5 minutes time-bins and hence it's not possible to determine for sure whether a record is contributing or not to a certain aggregate. Only work-around i can think of for your case is to permanenly enable debug in pmacct (or logging in the RDBMS) so that it's possible, at any moment, to know which queries have been performed. This at the expense of slightly more resources. If using default SQL schemas and an INSERT-only scenario, to protect against duplicates, you could have simply appended the '-i' option to pmpgplay to disable UPDATE SQL queries. Logics is explained here: So I'd be better off making these 2 match? sql_refresh_time: 60 sql_history: 5m (whether it's 60 secs or 5m's, they should be the same). For what it's worth the log seems to be aggregated on the history time. I only see the 5 min increments when I run pmpgplay in debug mode. -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Chris, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Your specific configuration is tricky because you write to the RDBMS every 60 seconds but aggregate on 5 minutes time-bins and hence it's not possible to determine for sure whether a record is contributing or not to a certain aggregate. Only work-around i can think of for your case is to permanenly enable debug in pmacct (or logging in the RDBMS) so that it's possible, at any moment, to know which queries have been performed. This at the expense of slightly more resources. If using default SQL schemas and an INSERT-only scenario, to protect against duplicates, you could have simply appended the '-i' option to pmpgplay to disable UPDATE SQL queries. Logics is explained here: http://wiki.pmacct.net/CustomizingTheSqlIndexes Cheers, Paolo On Fri, Jul 23, 2010 at 11:53:53AM +1000, Chris wrote: Hi, I'm using pmacct-0.12.0rc3 with postgresql. I've been trying to work out what pmpgplay does when it looks at a log file. I couldn't find much in my searches that go into the details of how it does things. It seems it generates a bunch of sql, but does it check to see if that record exists already before applying the change? I'm worried about replaying the log and it double-recording data (ie the data is already recorded in the db, and then replaying the log does it again), or will it only apply new records that it can't find in the db? I have a missing chunk of data from my database (about 1.5 hrs worth) that I can see in the logs, but the log is surrounded by data that could be in the database (I haven't checked each individual log entry). Relevant pmacct.conf info: plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 sql_table_version:1 sql_refresh_time: 60 sql_history: 5m Any information is much appreciated, thanks! -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmpgplay - what does it do?
Hi, I'm using pmacct-0.12.0rc3 with postgresql. I've been trying to work out what pmpgplay does when it looks at a log file. I couldn't find much in my searches that go into the details of how it does things. It seems it generates a bunch of sql, but does it check to see if that record exists already before applying the change? I'm worried about replaying the log and it double-recording data (ie the data is already recorded in the db, and then replaying the log does it again), or will it only apply new records that it can't find in the db? I have a missing chunk of data from my database (about 1.5 hrs worth) that I can see in the logs, but the log is surrounded by data that could be in the database (I haven't checked each individual log entry). Relevant pmacct.conf info: plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 sql_table_version:1 sql_refresh_time: 60 sql_history: 5m Any information is much appreciated, thanks! -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
