Re: [pmacct-discussion] Graphing Options

[Robert Juric]

I wanted to check back in and say thanks to everyone. After some time
digging in I finally got my nfacctd data into InfluxDB and Grafana. Had to
learn a lot on the way but I can finally put nfacctd to good use now!

Robert


On Thu, Nov 10, 2016 at 5:07 AM, Rasto Rickardt  wrote:

> Robert,
>
> Generally - if you want Top 5 talkers in some time range, you need to
> store it somewhere to be able to select them. Both works in SQL DB and
> NoSQL - time series DB. RRD based solution will not give you features
> you need.
>
> If you speak about conversation, i suppose aggregation on
> src_host/dst_host you are interested in for a time range.
>
> You have it from memory, or f.e AMQP plugin.
>
> You have output like
> src inB outB
> 1.2.3.4 100 200
>
> You can push it to InfluxDB like this:
>
> dbname,src=$src inputbytes=$inB,outputbytes=$outB
> timestamp is added automatically from the time of insert.
>
> Grafana query can look like this
>
> SELECT (last("inputbytes") - first("inputbytes")) + (last("outputbytes")
> - first("outputbytes")) as "data" FROM "dbname" WHERE $timeFilter  and
> "src" =~ /$src$/ GROUP BY timestamp
>
> $timeFilter is time window selected in Grafana to show (15 minutes, 30
> days)
>
> It is possible that i do not fully understand your use case, but i hope
> this will help you to have an idea how to do this.
>
> r.
>
>
> On 11/09/2016 11:55 PM, Robert Juric wrote:
> > What I'm not sure of is whether or not time-series is the correct way to
> > store my data? I am currently aggregating nfacctd data based on flow
> > timestamps for accounting purposes. For those using InfuxDB and
> > Graphite/Graphana, what primitives are you aggregating on and what do
> > you pull out of the data in the graphing solution?
> >
> >
> >
> > On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  > > wrote:
> >
> > I would use InfluxDB as database & Grafana for graphing.
> >
> > As you already using memory plugin, you can use pmacct client and
> push
> > data to InfluxDB. It is webservice, so simple bash & curl will work.
> >
> > r.
> >
> > On 11/09/2016 11:01 PM, Robert Juric wrote:
> > > After fiddling around for a few days I'm still at a loss for
> finding a
> > > good graphing option.
> > >
> > > I've been working today trying to use the memory plugin and cacti
> to
> > > graph some data, but I realized that it won't be good for dynamic
> type
> > > graphs. I could easily graph total tcp/udp traffic since those
> don't
> > > alter too much. Ideally I'd like to aggregate the dst_ports but
> > I'd like
> > > to graph the top 5. I understand Cacti may not be the best for this
> > > since you have to define the data sources manually.
> > >
> > > What other options are available or commonly used for graphing?
> > > Preferably something that can be aggregated on a per conversation
> > basis?
> > >
> > >
> > > ___
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> > 
> > >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> > 
> >
> >
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

[Catalin Petrescu]

Hi Raphael,

We are doing one measurment for each netflow exporter. Also we keep only
the data for 60 days ( retention policy on influxdb ). On top of that we
run continuous queries to generate a new measurment for total traffic in/
total traffic in from transit/peers , this are the most used graphs by our
noc.
Even with above influxdb still uses ~10G of ram constant. I beleive this
can be optimised if you don't use that many primitives and/or use less
tags.

Ram is cheap anyway so for us it does the job.

Regards,
Catalin

On Thu, Nov 10, 2016 at 8:43 PM, Raphael Mazelier  wrote:

>
>
> On 10/11/2016 20:58, Catalin Petrescu wrote:
>
>> Hi Robert,
>>
>> We had good results with grafana and influxdb using below  :
>> -src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,
>> dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path
>>
>> We export the flow records from pmacctd to kafka then import into influxdb
>> and from there is just a matter of creating the templates.  The only
>> custom
>> thing is kafka to influx , and we use kafka-influxdb with a inhouse
>> parser *
>> (kafka-influxdb). * take a look
>>
>> * here
>> > analysis-tool.html>*
>>
>>
>>
> Very interesting. I've made something similar using custom amqp consumer.
> Is influxdb support well as much data ? because you basically insert one ts
> point for one flow entry.
>
> --
> Raphael Mazelier
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

[Raphael Mazelier]

On 10/11/2016 20:58, Catalin Petrescu wrote:

Hi Robert,

We had good results with grafana and influxdb using below  :
-src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path

We export the flow records from pmacctd to kafka then import into influxdb
and from there is just a matter of creating the templates.  The only custom
thing is kafka to influx , and we use kafka-influxdb with a inhouse parser *
(kafka-influxdb). * take a look

* here
*




Very interesting. I've made something similar using custom amqp 
consumer. Is influxdb support well as much data ? because you basically 
insert one ts point for one flow entry.


--
Raphael Mazelier

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

[Catalin Petrescu]

Hi Robert,

We had good results with grafana and influxdb using below  :
-src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path

We export the flow records from pmacctd to kafka then import into influxdb
and from there is just a matter of creating the templates.  The only custom
thing is kafka to influx , and we use kafka-influxdb with a inhouse parser *
(kafka-influxdb). * take a look

* here
*

On Wed, Nov 9, 2016 at 10:55 PM, Robert Juric 
wrote:

> What I'm not sure of is whether or not time-series is the correct way to
> store my data? I am currently aggregating nfacctd data based on flow
> timestamps for accounting purposes. For those using InfuxDB and
> Graphite/Graphana, what primitives are you aggregating on and what do you
> pull out of the data in the graphing solution?
>
>
>
> On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  wrote:
>
>> I would use InfluxDB as database & Grafana for graphing.
>>
>> As you already using memory plugin, you can use pmacct client and push
>> data to InfluxDB. It is webservice, so simple bash & curl will work.
>>
>> r.
>>
>> On 11/09/2016 11:01 PM, Robert Juric wrote:
>> > After fiddling around for a few days I'm still at a loss for finding a
>> > good graphing option.
>> >
>> > I've been working today trying to use the memory plugin and cacti to
>> > graph some data, but I realized that it won't be good for dynamic type
>> > graphs. I could easily graph total tcp/udp traffic since those don't
>> > alter too much. Ideally I'd like to aggregate the dst_ports but I'd like
>> > to graph the top 5. I understand Cacti may not be the best for this
>> > since you have to define the data sources manually.
>> >
>> > What other options are available or commonly used for graphing?
>> > Preferably something that can be aggregated on a per conversation basis?
>> >
>> >
>> > ___
>> > pmacct-discussion mailing list
>> > http://www.pmacct.net/#mailinglists
>> >
>>
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

[Robert Juric]

What I'm not sure of is whether or not time-series is the correct way to
store my data? I am currently aggregating nfacctd data based on flow
timestamps for accounting purposes. For those using InfuxDB and
Graphite/Graphana, what primitives are you aggregating on and what do you
pull out of the data in the graphing solution?



On Wed, Nov 9, 2016 at 4:21 PM, Rasto Rickardt  wrote:

> I would use InfluxDB as database & Grafana for graphing.
>
> As you already using memory plugin, you can use pmacct client and push
> data to InfluxDB. It is webservice, so simple bash & curl will work.
>
> r.
>
> On 11/09/2016 11:01 PM, Robert Juric wrote:
> > After fiddling around for a few days I'm still at a loss for finding a
> > good graphing option.
> >
> > I've been working today trying to use the memory plugin and cacti to
> > graph some data, but I realized that it won't be good for dynamic type
> > graphs. I could easily graph total tcp/udp traffic since those don't
> > alter too much. Ideally I'd like to aggregate the dst_ports but I'd like
> > to graph the top 5. I understand Cacti may not be the best for this
> > since you have to define the data sources manually.
> >
> > What other options are available or commonly used for graphing?
> > Preferably something that can be aggregated on a per conversation basis?
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Graphing Options

[Cameron Murray]

This was actually being discussed yesterday on AUSNOG. A recommendation was
http://uowits.github.io/herbert-gui/index.html however more suited to end
user billing it might be suited to your needs?

On Thu, Nov 10, 2016 at 8:01 AM, Robert Juric 
wrote:

> After fiddling around for a few days I'm still at a loss for finding a
> good graphing option.
>
> I've been working today trying to use the memory plugin and cacti to graph
> some data, but I realized that it won't be good for dynamic type graphs. I
> could easily graph total tcp/udp traffic since those don't alter too much.
> Ideally I'd like to aggregate the dst_ports but I'd like to graph the top
> 5. I understand Cacti may not be the best for this since you have to define
> the data sources manually.
>
> What other options are available or commonly used for graphing? Preferably
> something that can be aggregated on a per conversation basis?
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists