Re: [pmacct-discussion] Redis support

2016-11-14 Thread Lorenzo Mainardi 
Logstash supports both Kafka both Rabbitmq as input 
(https://www.elastic.co/guide/en/logstash/current/input-plugins.html)


Telegraf does not (https://docs.influxdata.com/telegraf/v1.1/inputs/)


Redis is fast (it's written in C), easy to setup and deploy and very easy to 
use as message queue.


Input plugins | Logstash Reference [5.0] | 
Elastic<https://www.elastic.co/guide/en/logstash/current/input-plugins.html>
www.elastic.co
An input plugin enables a specific source of events to be read by Logstash. The 
following input plugins are available:



Da: pmacct-discussion  per conto di Paolo 
Lucente 
Inviato: lunedì 14 novembre 2016 18.02.19
A: pmacct-discussion@pmacct.net
Oggetto: Re: [pmacct-discussion] Redis support


Hi Lorenzo,

You are the second person asking for this - i'd be curious to see on the
list if there would be anybody else interested in this.

I have a question, maybe basic: would not Kafka or RabbitMQ be integrated
the same way with Logstash and/or Telegraf?

Cheers,
Paolo

On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
> Hi Paolo,
> I'm writing to asking if you are thinking about supporting Redis as plugin
>
> The advantage to use Redis would be that both Logstash both Telegraf support 
> it in a native way, then it would be easy to store pmacct data in native way 
> in Elasticsearch or InfluxDB.
> Regards
>
>
> digitel
>
> Via della Fortezza 6 - 50129 Firenze
> www.digitelitalia.com<http://www.digitelitalia.com/<http://www.digitelitalia.com<http://www.digitelitalia.com/>>
>  - 800 901 669
>
> Ing. Lorenzo Mainardi
>
> Tel +39 055 4624933
> Fax +39 055 4624 947
> l...@digitelitalia.com<mailto:l...@digitelitalia.com>
>
>

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

2016-11-14 Thread Raphael Mazelier 



On 14/11/2016 19:42, Rasto Rickardt wrote:

Hello Paolo,

+1 for me, main reasons are the usual Redis ones:



Pmacct have already a lot of backend plugins, but if another is needed I 
vote for redis. Redis is blazing fast and can act both as a message 
queue and a an ephemeral storage. I think I can use it for live 
data/trends, and passing the history to influx ?


With sql/amqp/kafka + redis pmacct will cover all the current 
technologies. (perhaps the mongodb plugins was not necessary ;)


--
Raphael Mazelier

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

2016-11-14 Thread Rasto Rickardt 
Hello Paolo,

+1 for me, main reasons are the usual Redis ones:

Speed - 100k GET by_id per second
Data persistence across restarts
Multi node deployment is not a MUST like other NoSQL document stores
Lightweight

My use-case is storing time-based events from flows in InfluxDB (bps,
pps, flows per host/subnet) and details (tcp flags, packet sizes,
as_path etc.) in redis. Storing timestamp, tag and value is where Influx
is good, storing documentation is where redis shines.

Storing directly to redis can be usefull, because most of data amount
per flow goes there for me.

The same will apply for InfluxDB but i am interfacing it over http, so
there is bunch of overhead already, i do not think there will be major
performance improvement writing directly, and tag:value pairs are really
lightweight.

r.


On 11/14/2016 06:02 PM, Paolo Lucente wrote:
> 
> Hi Lorenzo,
> 
> You are the second person asking for this - i'd be curious to see on
> the list if there would be anybody else interested in this.
> 
> I have a question, maybe basic: would not Kafka or RabbitMQ be
> integrated the same way with Logstash and/or Telegraf?
> 
> Cheers, Paolo
> 
> On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
>> Hi Paolo, I'm writing to asking if you are thinking about
>> supporting Redis as plugin
>> 
>> The advantage to use Redis would be that both Logstash both
>> Telegraf support it in a native way, then it would be easy to store
>> pmacct data in native way in Elasticsearch or InfluxDB. Regards
>> 
>> 
>> digitel
>> 
>> Via della Fortezza 6 - 50129 Firenze 
>> www.digitelitalia.com - 800 901 669
>> 
>> Ing. Lorenzo Mainardi
>> 
>> Tel +39 055 4624933 Fax +39 055 4624 947 
>> l...@digitelitalia.com
>> 
>> 
> 
>> ___ pmacct-discussion
>> mailing list http://www.pmacct.net/#mailinglists
> 
> 
> ___ pmacct-discussion
> mailing list http://www.pmacct.net/#mailinglists
> 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

2016-11-14 Thread Paolo Lucente 

Hi Lorenzo,

You are the second person asking for this - i'd be curious to see on the
list if there would be anybody else interested in this.

I have a question, maybe basic: would not Kafka or RabbitMQ be integrated
the same way with Logstash and/or Telegraf?

Cheers,
Paolo
 
On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
> Hi Paolo,
> I'm writing to asking if you are thinking about supporting Redis as plugin
> 
> The advantage to use Redis would be that both Logstash both Telegraf support 
> it in a native way, then it would be easy to store pmacct data in native way 
> in Elasticsearch or InfluxDB.
> Regards
> 
> 
> digitel
> 
> Via della Fortezza 6 - 50129 Firenze
> www.digitelitalia.com - 800 901 669
> 
> Ing. Lorenzo Mainardi
> 
> Tel +39 055 4624933
> Fax +39 055 4624 947
> l...@digitelitalia.com
> 
> 

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Redis support

2016-11-14 Thread Lorenzo Mainardi 
Hi Paolo,
I'm writing to asking if you are thinking about supporting Redis as plugin

The advantage to use Redis would be that both Logstash both Telegraf support it 
in a native way, then it would be easy to store pmacct data in native way in 
Elasticsearch or InfluxDB.
Regards


digitel

Via della Fortezza 6 - 50129 Firenze
www.digitelitalia.com - 800 901 669

Ing. Lorenzo Mainardi

Tel +39 055 4624933
Fax +39 055 4624 947
l...@digitelitalia.com


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists