Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 03/08/2016 à 14:43, Catalin Petrescu a écrit :


Why passing by telegraf ? is a kafka (or amqp) consumer not enough ?



telegraf is acting as kafka consumer in this case.



Sure, but what about writing your own (or use kafka-influxdb with 
specific encoder) ? anyway this is a creative way of using telegraf :)


--
Raphael Mazelier


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Karl O. Pinc]

On Wed, 3 Aug 2016 14:31:08 +0200
raf  wrote:

> Le 02/08/2016 à 17:41, Karl O. Pinc a écrit :

> > And rrd-tool seems
> > like the tool for that job.
> >  
> 
> rdd still made a great job, but I think there are better option today.
> (influx/graphite/grafana).

Thanks.  I've not been keeping up.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Catalin Petrescu]

On Wed, Aug 3, 2016 at 1:07 PM, raf  wrote:

>
> Why passing by telegraf ? is a kafka (or amqp) consumer not enough ?
>
>
>
telegraf is acting as kafka consumer in this case.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 02/08/2016 à 17:41, Karl O. Pinc a écrit :



Time-series storage seems the way to go.


Yes for graphing. For tabular or analytic traditionnals sgbd are more 
suited for me.



And rrd-tool seems
like the tool for that job.



rdd still made a great job, but I think there are better option today.
(influx/graphite/grafana).

--
Raphael Mazelier


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 02/08/2016 à 17:35, Robert Juric a écrit :

Well would anyone else be interested in developing a dedicated front-end
utilizing the existingpmacct database? Or is it the general consensus
that everyone exports the pmacct data to other systems for graphical
representation?




The problem is everyone want something different.
I ve made mine but it was really oriented on my buisness need.
It is way more easy to pass to export (or consolidate) metrics on a 
general purpose metric system in my opinion (graphite/influx/whatever 
for the tsdb), grafana or other for the gui.



--
Raphael Mazelier



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 03/08/2016 à 13:25, Catalin Petrescu a écrit :

hi ,

pmacctd->kafka->telegraf->influxdb->grafana works great for me if used
for peering analitics.



Why passing by telegraf ? is a kafka (or amqp) consumer not enough ?

--
Raphael Mazelier


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Catalin Petrescu]

hi ,

pmacctd->kafka->telegraf->influxdb->grafana works great for me if used for
peering analitics.

http://imgur.com/a/u2OrM

Unfurtunatlly because i can't get telegraf to make tags on fields that are
not strings i have to rely on a script to get the data from
influxdb(measurment1) to influxdb ( measurment2 with tags).

Let me know if that helps. More than happy to share the scripts and
templates.

Regards,
Catalin

On Tue, Aug 2, 2016 at 5:56 PM, Karl O. Pinc  wrote:

> On Tue, 02 Aug 2016 17:59:14 +0200
> Davide Principi  wrote:
>
> > On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote:
> > > Time-series storage seems the way to go.  And rrd-tool seems
> > > like the tool for that job.
> >
> > What if accounting for any IP of the network is required? Wouldn't it
> > require too much disk space?
>
> Perhaps.  For some value of "too much".  But really, the point
> of rrdtool is that you say how much data you want to store
> and the space is pre-allocated and it never gets any bigger.
> You just increasingly lose resolution as a data point ages.
> Sorta the point of time-series storage.
>
>
> Karl 
> Free Software:  "You don't pay back, you pay forward."
>  -- Robert A. Heinlein
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Karl O. Pinc]

On Tue, 02 Aug 2016 17:59:14 +0200
Davide Principi  wrote:

> On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote:
> > Time-series storage seems the way to go.  And rrd-tool seems
> > like the tool for that job.  
> 
> What if accounting for any IP of the network is required? Wouldn't it
> require too much disk space?

Perhaps.  For some value of "too much".  But really, the point
of rrdtool is that you say how much data you want to store
and the space is pre-allocated and it never gets any bigger.
You just increasingly lose resolution as a data point ages.
Sorta the point of time-series storage.


Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Davide Principi]

On Tue, 2016-08-02 at 10:41 -0500, Karl O. Pinc wrote:
> Time-series storage seems the way to go.  And rrd-tool seems
> like the tool for that job.

What if accounting for any IP of the network is required? Wouldn't it
require too much disk space?



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Karl O. Pinc]

On Tue, 2 Aug 2016 10:35:44 -0500
Robert Juric  wrote:

> Well would anyone else be interested in developing a dedicated
> front-end utilizing the existingpmacct database? Or is it the general
> consensus that everyone exports the pmacct data to other systems for
> graphical representation?

The approach I'd (often) choose is to use a memory plugin to feed
data to rrdtool.  And then use mtrg-rrd to generate graphs.

Time-series storage seems the way to go.  And rrd-tool seems
like the tool for that job.

I keep thinking I'll write something that "automates" this setup,
particularly the feeding into rrdtool.  But I never do.



Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Robert Juric]

Well would anyone else be interested in developing a dedicated front-end
utilizing the existingpmacct database? Or is it the general consensus that
everyone exports the pmacct data to other systems for graphical
representation?



On Tue, Aug 2, 2016 at 8:40 AM, Davide Principi  wrote:

> On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote:
> > I'm looking for a bandwidthd replacement and I started experimenting
> > with pmacct.
>
> Well thanks again guys for all your suggestions!
>
> Just for the record, I decided to enable the sqlite backend on
> bandwidthd, by compiling it with an old patch starting from the Fedora
> RPM.
>
> You know, my customers are happy with its interface and it's hard to
> find a good substitute.
>
> Source code is available here:
> https://github.com/NethServer/bandwidthd
>
> --
> Davide Principi
>
> #davidep | @davideprincipi | GPG 0x5651EA71
>
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Davide Principi]

On Tue, 2016-07-26 at 15:39 +0200, Davide Principi wrote:
> I'm looking for a bandwidthd replacement and I started experimenting
> with pmacct.  

Well thanks again guys for all your suggestions! 

Just for the record, I decided to enable the sqlite backend on
bandwidthd, by compiling it with an old patch starting from the Fedora
RPM.

You know, my customers are happy with its interface and it's hard to
find a good substitute.

Source code is available here:
https://github.com/NethServer/bandwidthd

-- 
Davide Principi

#davidep | @davideprincipi | GPG 0x5651EA71




___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 29/07/2016 à 09:03, Davide Principi a écrit :



Thanks raf! This seems more comfortable!


My php gui use mysql as backend, but it is trivial to change to sqlite 
(or whatever).




Does it produce any graph? Could you attach some screenshots?



No. I graph with grafana.
But if you carefully choose what metric you put in your backend (aka not 
storing full flows), the db should remain small, and it will relatively 
easy to wrote a grapher with whatever lib (highchart is cool).


You can also look at sir (https://github.com/dbarrosop/sir) if you like 
python, it use sqlite as backend afaik.


--
Raphael Mazelier

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 28/07/2016 à 08:50, Linas Lesauskas a écrit :

Hi,

just my 2 cents.

I found very useful InfluxDB (https://influxdata.com/) for time-series
data storage. It is extremely fast and lean on storage questions, uses
kind of 20 bytes per record.
Combining with Grafana (http://grafana.org/) you can build classy and
stylish user interface.

Currently we are playing with pmacct->bunch of scripts->influx on the
lab and waiting for a more free time to push this to production.

best regards,



I've used both approach :

pmacct > amqp > scripts > influxdb > grafana to display some nice graphs.

and custom home made gui (in php :/ ) to display specific informations 
(top 10 interface / asn / as path / talkers - in/out).


All this material is available on https://github.com/ut0mt8 .

I think there are trivial to adapt on your needs.

--
Raphael Mazelier


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[Robert Juric]

I had started to work with HighCharts to put a front-end together for my
small deployment. I also wouldn't mind contributing to a project in any way
I could.

Robert Juric

On Tue, Jul 26, 2016 at 9:58 AM, Davide Principi <
davide.princ...@nethesis.it> wrote:

> Thanks for the prompt reply, Harry!
>
> >
> > You might be interested in: http://uowits.github.io/herbert-gui/index
> > .html
>
>
> It looks great, but if I understand correctly that UI requires MongoDB
> and RabbitMQ messaging queue to collect data.  Of course, I would not
> run that infrastructure on a single router!
>
> Any other idea?
>
> >
> > One thing you might notice is that due to the flexible nature of
> > pmacct, creating an all encompassing front-end is quite a mammoth
> > task. I think a lot of people tend to plug the aggregates into their
> > existing infrastructure.
>
> This is an important point! It could also explain why a simple UI that
> runs on the same host where data is collected is not so easy to
> find...
>
> Nobody is interested on a similar project in the "datacenter era" :) ?
>
> However SME businesses, non-profit orgs with a LAN and their firewall
> could appreciate it... What do you think?
>
> >
> > That being said, I'd be happy to help contribute if you do decide to
> > start a project.
> >
>
> This is awesome, I'll keep you informed!
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Looking for a fresh pmacct UI

[Davide Principi]

I'm looking for a bandwidthd replacement and I started experimenting
with pmacct.  The monitoring daemon and the cli tool are amazing and
seem to fit my needs;  however, on the web site links to external tools
for displaying data are quite old.

My ideal UI should be helpful on finding anomalous traffic from LAN.
Requirements could be

- small network monitoring (<1000 nodes) 
- storage on local file(s) like, sqlite, csv, json (no RDBMS!)
- counters by IP, proto (ports, maybe)
- periodical graphs
- list of "top talkers"

Could you point me to other projects providing an UI for pmacct? 

Is there anybody interested on such project?

Meanwhile I built a pmacct RPM for NethServer/CentOS 7.  You can find
it on

http://packages.nethserver.org/nethserver/7.2.1511/nethforge-testin
g/x86_64/Packages/pmacct-1.6.0-1.ns7.x86_64.rpm

initial .spec file on GitHub

https://github.com/DavidePrincipi/pmacct

NethServer is the community project behind my initiative, please come
to visit us on http://community.nethserver.org.

Best regards,

-- 
Davide Principi

#davidep | @davideprincipi | GPG 0x5651EA71




___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists