Re: [pmacct-discussion] Looking for a fresh pmacct UI
On Tue, 02 Aug 2016 17:59:14 +0200 Davide Principi wrote: > On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote: > > Time-series storage seems the way to go. And rrd-tool seems > > like the tool for that job. > > What if accounting for any IP of the network is required? Wouldn't it > require too much disk space? Perhaps. For some value of "too much". But really, the point of rrdtool is that you say how much data you want to store and the space is pre-allocated and it never gets any bigger. You just increasingly lose resolution as a data point ages. Sorta the point of time-series storage. Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Looking for a fresh pmacct UI
On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote: > Time-series storage seems the way to go. And rrd-tool seems > like the tool for that job. What if accounting for any IP of the network is required? Wouldn't it require too much disk space? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Looking for a fresh pmacct UI
On Tue, 2 Aug 2016 10:35:44 -0500 Robert Juric wrote: > Well would anyone else be interested in developing a dedicated > front-end utilizing the existingpmacct database? Or is it the general > consensus that everyone exports the pmacct data to other systems for > graphical representation? The approach I'd (often) choose is to use a memory plugin to feed data to rrdtool. And then use mtrg-rrd to generate graphs. Time-series storage seems the way to go. And rrd-tool seems like the tool for that job. I keep thinking I'll write something that "automates" this setup, particularly the feeding into rrdtool. But I never do. Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Looking for a fresh pmacct UI
Well would anyone else be interested in developing a dedicated front-end utilizing the existingpmacct database? Or is it the general consensus that everyone exports the pmacct data to other systems for graphical representation? On Tue, Aug 2, 2016 at 8:40 AM, Davide Principi wrote: > On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote: > > I'm looking for a bandwidthd replacement and I started experimenting > > with pmacct. > > Well thanks again guys for all your suggestions! > > Just for the record, I decided to enable the sqlite backend on > bandwidthd, by compiling it with an old patch starting from the Fedora > RPM. > > You know, my customers are happy with its interface and it's hard to > find a good substitute. > > Source code is available here: > https://github.com/NethServer/bandwidthd > > -- > Davide Principi > > #davidep | @davideprincipi | GPG 0x5651EA71 > > > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Fwd: minb - version 1.5.3
Hi Paolo, I had to look at the code but I figured it out. needed minb= not minb>= On 08/02/2016 09:36 AM, Steve Clark wrote: Hi Paolo, I am trying to limit netflow aggregates to greater than 100 bytes before insertion into my PG database, but I can't seem to get it to work. All aggregates are being inserted. my config: daemonize: true debug: false pidfile: /var/run/nfacctd.pid syslog: daemon pre_tag_map: ./my.pretag.map nfacctd_disable_checks: true nfacctd_time_new: false aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos plugin_pipe_size: 4096000 plugin_buffer_size: 4096 plugins: pgsql sql_table: netflow sql_data: typed sql_dont_try_update: true sql_use_copy: true sql_db: pmacct sql_host: 127.0.0.1 sql_passwd: arealsmartpwd sql_user: pmacct sql_refresh_time: 60 sql_optimize_clauses: true sql_history: 1m sql_history_roundoff: m sql_recovery_logfile: /var/lib/pmacct/recovery_log sql_preprocess: minb>=100 sql_locking_style: row sql_cache_entries: 19 imt_buckets: 65537 imt_mem_pools_size: 1024000 nfacctd_port: 2055 Here is what I get in my table - notice 1173 under 100 bytes. pmacct=# truncate netflow ; TRUNCATE TABLE pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow where agent_id = '246' and bytes <100; count | bytes | packets ---+---+- 1173 | 89321 |1205 (1 row) pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow where agent_id = '246'; count | bytes | packets ---+--+- 3690 | 63424928 | 105921 (1 row) Also this is what shows from /var/log/messages Aug 2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache - START (PID: 4073) *** Aug 2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache - END (PID: 4073, QN: 3690/3690, ET: 0) *** KEY: minb DESC: check. Aggregates on the queue are evaluated one-by-one; each object is marked valid only if the bytes counter is '>=' minb value. An interesting idea is to set its value to a fraction of the link capacity. Remember that you have also a timeframe reference: the 'sql_refresh_time' seconds. All plugins. For example, given the following parameters: Link Capacity = 8Mbit/s, THreshold = 0.1%, TImeframe = 60s minb = ((LC / 8) * TI) * TH -> ((8Mbit/s / 8) * 60s) * 0.1% = 6 bytes. Given a 8Mbit link, all aggregates which have accounted for at least 60Kb of traffic in the last 60 seconds, will be written to the DB. Any suggestions? Thanks, Steve ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson) ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Looking for a fresh pmacct UI
On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote: > I'm looking for a bandwidthd replacement and I started experimenting > with pmacct. Well thanks again guys for all your suggestions! Just for the record, I decided to enable the sqlite backend on bandwidthd, by compiling it with an old patch starting from the Fedora RPM. You know, my customers are happy with its interface and it's hard to find a good substitute. Source code is available here: https://github.com/NethServer/bandwidthd -- Davide Principi #davidep | @davideprincipi | GPG 0x5651EA71 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Fwd: minb - version 1.5.3
Hi Paolo, I am trying to limit netflow aggregates to greater than 100 bytes before insertion into my PG database, but I can't seem to get it to work. All aggregates are being inserted. my config: daemonize: true debug: false pidfile: /var/run/nfacctd.pid syslog: daemon pre_tag_map: ./my.pretag.map nfacctd_disable_checks: true nfacctd_time_new: false aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos plugin_pipe_size: 4096000 plugin_buffer_size: 4096 plugins: pgsql sql_table: netflow sql_data: typed sql_dont_try_update: true sql_use_copy: true sql_db: pmacct sql_host: 127.0.0.1 sql_passwd: arealsmartpwd sql_user: pmacct sql_refresh_time: 60 sql_optimize_clauses: true sql_history: 1m sql_history_roundoff: m sql_recovery_logfile: /var/lib/pmacct/recovery_log sql_preprocess: minb>=100 sql_locking_style: row sql_cache_entries: 19 imt_buckets: 65537 imt_mem_pools_size: 1024000 nfacctd_port: 2055 Here is what I get in my table - notice 1173 under 100 bytes. pmacct=# truncate netflow ; TRUNCATE TABLE pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow where agent_id = '246' and bytes <100; count | bytes | packets ---+---+- 1173 | 89321 |1205 (1 row) pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow where agent_id = '246'; count | bytes | packets ---+--+- 3690 | 63424928 | 105921 (1 row) Also this is what shows from /var/log/messages Aug 2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache - START (PID: 4073) *** Aug 2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache - END (PID: 4073, QN: 3690/3690, ET: 0) *** KEY: minb DESC: check. Aggregates on the queue are evaluated one-by-one; each object is marked valid only if the bytes counter is '>=' minb value. An interesting idea is to set its value to a fraction of the link capacity. Remember that you have also a timeframe reference: the 'sql_refresh_time' seconds. All plugins. For example, given the following parameters: Link Capacity = 8Mbit/s, THreshold = 0.1%, TImeframe = 60s minb = ((LC / 8) * TI) * TH -> ((8Mbit/s / 8) * 60s) * 0.1% = 6 bytes. Given a 8Mbit link, all aggregates which have accounted for at least 60Kb of traffic in the last 60 seconds, will be written to the DB. Any suggestions? Thanks, Steve ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists