Re: [pmacct-discussion] Graphing Options

2016-11-24 Thread Robert Juric 
I wanted to check back in and say thanks to everyone. After some time
digging in I finally got my nfacctd data into InfluxDB and Grafana. Had to
learn a lot on the way but I can finally put nfacctd to good use now!

Robert


On Thu, Nov 10, 2016 at 5:07 AM, Rasto Rickardt  wrote:

> Robert,
>
> Generally - if you want Top 5 talkers in some time range, you need to
> store it somewhere to be able to select them. Both works in SQL DB and
> NoSQL - time series DB. RRD based solution will not give you features
> you need.
>
> If you speak about conversation, i suppose aggregation on
> src_host/dst_host you are interested in for a time range.
>
> You have it from memory, or f.e AMQP plugin.
>
> You have output like
> src inB outB
> 1.2.3.4 100 200
>
> You can push it to InfluxDB like this:
>
> dbname,src=$src inputbytes=$inB,outputbytes=$outB
> timestamp is added automatically from the time of insert.
>
> Grafana query can look like this
>
> SELECT (last("inputbytes") - first("inputbytes")) + (last("outputbytes")
> - first("outputbytes")) as "data" FROM "dbname" WHERE $timeFilter  and
> "src" =~ /$src$/ GROUP BY timestamp
>
> $timeFilter is time window selected in Grafana to show (15 minutes, 30
> days)
>
> It is possible that i do not fully understand your use case, but i hope
> this will help you to have an idea how to do this.
>
> r.
>
>
> On 11/09/2016 11:55 PM, Robert Juric wrote:
> > What I'm not sure of is whether or not time-series is the correct way to
> > store my data? I am currently aggregating nfacctd data based on flow
> > timestamps for accounting purposes. For those using InfuxDB and
> > Graphite/Graphana, what primitives are you aggregating on and what do
> > you pull out of the data in the graphing solution?
> >
> >
> >
> > On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  > > wrote:
> >
> > I would use InfluxDB as database & Grafana for graphing.
> >
> > As you already using memory plugin, you can use pmacct client and
> push
> > data to InfluxDB. It is webservice, so simple bash & curl will work.
> >
> > r.
> >
> > On 11/09/2016 11:01 PM, Robert Juric wrote:
> > > After fiddling around for a few days I'm still at a loss for
> finding a
> > > good graphing option.
> > >
> > > I've been working today trying to use the memory plugin and cacti
> to
> > > graph some data, but I realized that it won't be good for dynamic
> type
> > > graphs. I could easily graph total tcp/udp traffic since those
> don't
> > > alter too much. Ideally I'd like to aggregate the dst_ports but
> > I'd like
> > > to graph the top 5. I understand Cacti may not be the best for this
> > > since you have to define the data sources manually.
> > >
> > > What other options are available or commonly used for graphing?
> > > Preferably something that can be aggregated on a per conversation
> > basis?
> > >
> > >
> > > ___
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> > 
> > >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> > 
> >
> >
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

2016-11-11 Thread Catalin Petrescu 
Hi Raphael,

We are doing one measurment for each netflow exporter. Also we keep only
the data for 60 days ( retention policy on influxdb ). On top of that we
run continuous queries to generate a new measurment for total traffic in/
total traffic in from transit/peers , this are the most used graphs by our
noc.
Even with above influxdb still uses ~10G of ram constant. I beleive this
can be optimised if you don't use that many primitives and/or use less
tags.

Ram is cheap anyway so for us it does the job.

Regards,
Catalin

On Thu, Nov 10, 2016 at 8:43 PM, Raphael Mazelier  wrote:

>
>
> On 10/11/2016 20:58, Catalin Petrescu wrote:
>
>> Hi Robert,
>>
>> We had good results with grafana and influxdb using below  :
>> -src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,
>> dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path
>>
>> We export the flow records from pmacctd to kafka then import into influxdb
>> and from there is just a matter of creating the templates.  The only
>> custom
>> thing is kafka to influx , and we use kafka-influxdb with a inhouse
>> parser *
>> (kafka-influxdb). * take a look
>>
>> * here
>> > analysis-tool.html>*
>>
>>
>>
> Very interesting. I've made something similar using custom amqp consumer.
> Is influxdb support well as much data ? because you basically insert one ts
> point for one flow entry.
>
> --
> Raphael Mazelier
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

2016-11-10 Thread Raphael Mazelier 



On 10/11/2016 20:58, Catalin Petrescu wrote:

Hi Robert,

We had good results with grafana and influxdb using below  :
-src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path

We export the flow records from pmacctd to kafka then import into influxdb
and from there is just a matter of creating the templates.  The only custom
thing is kafka to influx , and we use kafka-influxdb with a inhouse parser *
(kafka-influxdb). * take a look

* here
*




Very interesting. I've made something similar using custom amqp 
consumer. Is influxdb support well as much data ? because you basically 
insert one ts point for one flow entry.


--
Raphael Mazelier

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

2016-11-10 Thread Catalin Petrescu 
Hi Robert,

We had good results with grafana and influxdb using below  :
-src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path

We export the flow records from pmacctd to kafka then import into influxdb
and from there is just a matter of creating the templates.  The only custom
thing is kafka to influx , and we use kafka-influxdb with a inhouse parser *
(kafka-influxdb). * take a look

* here
*

On Wed, Nov 9, 2016 at 10:55 PM, Robert Juric 
wrote:

> What I'm not sure of is whether or not time-series is the correct way to
> store my data? I am currently aggregating nfacctd data based on flow
> timestamps for accounting purposes. For those using InfuxDB and
> Graphite/Graphana, what primitives are you aggregating on and what do you
> pull out of the data in the graphing solution?
>
>
>
> On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  wrote:
>
>> I would use InfluxDB as database & Grafana for graphing.
>>
>> As you already using memory plugin, you can use pmacct client and push
>> data to InfluxDB. It is webservice, so simple bash & curl will work.
>>
>> r.
>>
>> On 11/09/2016 11:01 PM, Robert Juric wrote:
>> > After fiddling around for a few days I'm still at a loss for finding a
>> > good graphing option.
>> >
>> > I've been working today trying to use the memory plugin and cacti to
>> > graph some data, but I realized that it won't be good for dynamic type
>> > graphs. I could easily graph total tcp/udp traffic since those don't
>> > alter too much. Ideally I'd like to aggregate the dst_ports but I'd like
>> > to graph the top 5. I understand Cacti may not be the best for this
>> > since you have to define the data sources manually.
>> >
>> > What other options are available or commonly used for graphing?
>> > Preferably something that can be aggregated on a per conversation basis?
>> >
>> >
>> > ___
>> > pmacct-discussion mailing list
>> > http://www.pmacct.net/#mailinglists
>> >
>>
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

2016-11-09 Thread Robert Juric 
What I'm not sure of is whether or not time-series is the correct way to
store my data? I am currently aggregating nfacctd data based on flow
timestamps for accounting purposes. For those using InfuxDB and
Graphite/Graphana, what primitives are you aggregating on and what do you
pull out of the data in the graphing solution?



On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  wrote:

> I would use InfluxDB as database & Grafana for graphing.
>
> As you already using memory plugin, you can use pmacct client and push
> data to InfluxDB. It is webservice, so simple bash & curl will work.
>
> r.
>
> On 11/09/2016 11:01 PM, Robert Juric wrote:
> > After fiddling around for a few days I'm still at a loss for finding a
> > good graphing option.
> >
> > I've been working today trying to use the memory plugin and cacti to
> > graph some data, but I realized that it won't be good for dynamic type
> > graphs. I could easily graph total tcp/udp traffic since those don't
> > alter too much. Ideally I'd like to aggregate the dst_ports but I'd like
> > to graph the top 5. I understand Cacti may not be the best for this
> > since you have to define the data sources manually.
> >
> > What other options are available or commonly used for graphing?
> > Preferably something that can be aggregated on a per conversation basis?
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

2016-11-09 Thread Cameron Murray 
This was actually being discussed yesterday on AUSNOG. A recommendation was
http://uowits.github.io/herbert-gui/index.html however more suited to end
user billing it might be suited to your needs?

On Thu, Nov 10, 2016 at 8:01 AM, Robert Juric 
wrote:

> After fiddling around for a few days I'm still at a loss for finding a
> good graphing option.
>
> I've been working today trying to use the memory plugin and cacti to graph
> some data, but I realized that it won't be good for dynamic type graphs. I
> could easily graph total tcp/udp traffic since those don't alter too much.
> Ideally I'd like to aggregate the dst_ports but I'd like to graph the top
> 5. I understand Cacti may not be the best for this since you have to define
> the data sources manually.
>
> What other options are available or commonly used for graphing? Preferably
> something that can be aggregated on a per conversation basis?
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Graphing Options

2016-11-09 Thread Robert Juric 
After fiddling around for a few days I'm still at a loss for finding a good
graphing option.

I've been working today trying to use the memory plugin and cacti to graph
some data, but I realized that it won't be good for dynamic type graphs. I
could easily graph total tcp/udp traffic since those don't alter too much.
Ideally I'd like to aggregate the dst_ports but I'd like to graph the top
5. I understand Cacti may not be the best for this since you have to define
the data sources manually.

What other options are available or commonly used for graphing? Preferably
something that can be aggregated on a per conversation basis?
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists