Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Karl O. Pinc]

On Tue, 02 Aug 2016 17:59:14 +0200
Davide Principi  wrote:

> On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote:
> > Time-series storage seems the way to go.  And rrd-tool seems
> > like the tool for that job.  
> 
> What if accounting for any IP of the network is required? Wouldn't it
> require too much disk space?

Perhaps.  For some value of "too much".  But really, the point
of rrdtool is that you say how much data you want to store
and the space is pre-allocated and it never gets any bigger.
You just increasingly lose resolution as a data point ages.
Sorta the point of time-series storage.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Davide Principi]

On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote:
> Time-series storage seems the way to go.  And rrd-tool seems
> like the tool for that job.

What if accounting for any IP of the network is required? Wouldn't it
require too much disk space?



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Karl O. Pinc]

On Tue, 2 Aug 2016 10:35:44 -0500
Robert Juric  wrote:

> Well would anyone else be interested in developing a dedicated
> front-end utilizing the existingpmacct database? Or is it the general
> consensus that everyone exports the pmacct data to other systems for
> graphical representation?

The approach I'd (often) choose is to use a memory plugin to feed
data to rrdtool.  And then use mtrg-rrd to generate graphs.

Time-series storage seems the way to go.  And rrd-tool seems
like the tool for that job.

I keep thinking I'll write something that "automates" this setup,
particularly the feeding into rrdtool.  But I never do.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Robert Juric]

Well would anyone else be interested in developing a dedicated front-end
utilizing the existingpmacct database? Or is it the general consensus that
everyone exports the pmacct data to other systems for graphical
representation?



On Tue, Aug 2, 2016 at 8:40 AM, Davide Principi  wrote:

> On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote:
> > I'm looking for a bandwidthd replacement and I started experimenting
> > with pmacct.
>
> Well thanks again guys for all your suggestions!
>
> Just for the record, I decided to enable the sqlite backend on
> bandwidthd, by compiling it with an old patch starting from the Fedora
> RPM.
>
> You know, my customers are happy with its interface and it's hard to
> find a good substitute.
>
> Source code is available here:
> https://github.com/NethServer/bandwidthd
>
> --
> Davide Principi
>
> #davidep | @davideprincipi | GPG 0x5651EA71
>
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Fwd: minb - version 1.5.3

[Stephen Clark]

Hi Paolo,

I had to look at the code but I figured it out.

needed minb= not minb>=

On 08/02/2016 09:36 AM, Steve Clark wrote:


Hi Paolo,

I am trying to limit netflow aggregates to greater than 100 bytes before 
insertion into my PG database, but

I can't seem to get it to work. All aggregates are being inserted.

my config:
daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon
pre_tag_map: ./my.pretag.map
nfacctd_disable_checks: true
nfacctd_time_new: false
aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos
plugin_pipe_size: 4096000
plugin_buffer_size: 4096
plugins: pgsql
sql_table: netflow
sql_data: typed
sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: arealsmartpwd
sql_user: pmacct
sql_refresh_time: 60
sql_optimize_clauses: true
sql_history: 1m
sql_history_roundoff: m
sql_recovery_logfile: /var/lib/pmacct/recovery_log
sql_preprocess: minb>=100
sql_locking_style: row
sql_cache_entries: 19
imt_buckets: 65537
imt_mem_pools_size: 1024000
nfacctd_port: 2055

Here is what I get in my table - notice 1173 under 100 bytes.
pmacct=# truncate netflow ;
TRUNCATE TABLE
pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from 
netflow where agent_id = '246' and bytes <100;

 count | bytes | packets
---+---+-
  1173 | 89321 |1205
(1 row)

pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from 
netflow where agent_id = '246';

count |  bytes   | packets
---+--+-
3690 | 63424928 |  105921
(1 row)

Also this is what shows from /var/log/messages
Aug  2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging 
cache - START (PID: 4073) ***
Aug  2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging 
cache - END (PID: 4073, QN: 3690/3690, ET: 0) ***


   KEY: minb
DESC: check. Aggregates on the queue are evaluated one-by-one; 
each object is marked valid
only if the bytes counter is '>=' minb value. An interesting idea is to set 
its value
  to a fraction of the link capacity. Remember that you 
have also a timeframe reference:

  the 'sql_refresh_time' seconds. All plugins.

  For example, given the following parameters:
  Link Capacity = 8Mbit/s, THreshold = 0.1%, TImeframe = 60s
  minb = ((LC / 8) * TI) * TH -> ((8Mbit/s / 8) * 60s) * 
0.1% = 6 bytes.


  Given a 8Mbit link, all aggregates which have accounted 
for at least 60Kb of traffic

  in the last 60 seconds, will be written to the DB.



Any suggestions?

Thanks,
Steve



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



--

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Davide Principi]

On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote:
> I'm looking for a bandwidthd replacement and I started experimenting
> with pmacct.  

Well thanks again guys for all your suggestions! 

Just for the record, I decided to enable the sqlite backend on
bandwidthd, by compiling it with an old patch starting from the Fedora
RPM.

You know, my customers are happy with its interface and it's hard to
find a good substitute.

Source code is available here:
https://github.com/NethServer/bandwidthd

-- 
Davide Principi

#davidep | @davideprincipi | GPG 0x5651EA71




___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Fwd: minb - version 1.5.3

[Steve Clark]

Hi Paolo,

I am trying to limit netflow aggregates to greater than 100 bytes before 
insertion into my PG database, but

I can't seem to get it to work. All aggregates are being inserted.

my config:
daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon
pre_tag_map: ./my.pretag.map
nfacctd_disable_checks: true
nfacctd_time_new: false
aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos
plugin_pipe_size: 4096000
plugin_buffer_size: 4096
plugins: pgsql
sql_table: netflow
sql_data: typed
sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: arealsmartpwd
sql_user: pmacct
sql_refresh_time: 60
sql_optimize_clauses: true
sql_history: 1m
sql_history_roundoff: m
sql_recovery_logfile: /var/lib/pmacct/recovery_log
sql_preprocess: minb>=100
sql_locking_style: row
sql_cache_entries: 19
imt_buckets: 65537
imt_mem_pools_size: 1024000
nfacctd_port: 2055

Here is what I get in my table - notice 1173 under 100 bytes.
pmacct=# truncate netflow ;
TRUNCATE TABLE
pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow 
where agent_id = '246' and bytes <100;

 count | bytes | packets
---+---+-
  1173 | 89321 |1205
(1 row)

pmacct=# select count(*),sum(bytes)as bytes,sum(packets)as packets from netflow 
where agent_id = '246';

count |  bytes   | packets
---+--+-
3690 | 63424928 | 105921
(1 row)

Also this is what shows from /var/log/messages
Aug  2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache 
- START (PID: 4073) ***
Aug  2 08:06:01 netflow nfacctd[4073]: INFO ( default/pgsql ): *** Purging cache 
- END (PID: 4073, QN: 3690/3690, ET: 0) ***


   KEY: minb
DESC: check. Aggregates on the queue are evaluated one-by-one; 
each object is marked valid
only if the bytes counter is '>=' minb value. An interesting idea is to set its 
value
  to a fraction of the link capacity. Remember that you 
have also a timeframe reference:

  the 'sql_refresh_time' seconds. All plugins.

  For example, given the following parameters:
  Link Capacity = 8Mbit/s, THreshold = 0.1%, TImeframe = 60s
  minb = ((LC / 8) * TI) * TH -> ((8Mbit/s / 8) * 60s) * 
0.1% = 6 bytes.


  Given a 8Mbit link, all aggregates which have accounted 
for at least 60Kb of traffic

  in the last 60 seconds, will be written to the DB.



Any suggestions?

Thanks,
Steve

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists