Re: [pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Dariush Marsh-Mossadeghi 
For that sort of volume I’d investigate using and ELK stack on the backend and 
de-coupling pmacct from storage and analysis with a message queue like Rabbit. 
I’ve used this approach (not to that scale though) and it works well.

ELK == ElasticSearch Logstash Kibana

> 
> 
> On 08/04/2016 11:48 AM, David McKen wrote:
>> For that type of scale maybe a SQL like NoSQL db like cassandra may work 
>> better for you.
>> 
>> On Thu, Aug 4, 2016 at 11:01 AM, Stephen Clark > > wrote:
>> Hi List,
>> 
>> I am looking to collect a large number of netflow records, on the order of a 
>> 100 million a day,
>> and store them in a postgres DB. Has anyone done this or something similar 
>> using pmacct?
>> 
>> Thanks,
>> Steve
>> 
>> 
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists 
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Stephen Clark 

Hmm...

I don't think pmacct directly support cassandra, but is does support  MongoDB.
Also I would like to be able to filter/sample the data at the point of origin, 
but
I don't think that is possible with pmacctd and the nfprobe module. It only 
seems
to be able to do it at the collector - nfacctd.

Steve

On 08/04/2016 11:48 AM, David McKen wrote:
For that type of scale maybe a SQL like NoSQL db like cassandra may work 
better for you.


On Thu, Aug 4, 2016 at 11:01 AM, Stephen Clark > wrote:


Hi List,

I am looking to collect a large number of netflow records, on the order of
a 100 million a day,
and store them in a postgres DB. Has anyone done this or something similar
using pmacct?

Thanks,
Steve


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists





--

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Stephen Clark 

Hi List,

I am looking to collect a large number of netflow records, on the order of a 100 
million a day,
and store them in a postgres DB. Has anyone done this or something similar using 
pmacct?


Thanks,
Steve


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists