Re: [pmacct-discussion] collecting large number of netflows

2016-08-19 Thread Jentsch, Mario
ussion-boun...@pmacct.net] > On Behalf Of Stephen Clark > Sent: Thursday, August 18, 2016 2:24 PM > To: pmacct-discussion@pmacct.net > Subject: Re: [pmacct-discussion] collecting large number of netflows > > On 08/17/2016 08:38 AM, Jentsch, Mario wrote: > > Hey Steve, > >

Re: [pmacct-discussion] collecting large number of netflows

2016-08-18 Thread Stephen Clark
m: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Stephen Clark Sent: Thursday, August 04, 2016 5:01 PM To: pmacct-discussion@pmacct.net Subject: [pmacct-discussion] collecting large number of netflows Hi List, I am looking to collect a large number of netflow records, on the or

Re: [pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Dariush Marsh-Mossadeghi
For that sort of volume I’d investigate using and ELK stack on the backend and de-coupling pmacct from storage and analysis with a message queue like Rabbit. I’ve used this approach (not to that scale though) and it works well. ELK == ElasticSearch Logstash Kibana > > > On 08/04/2016 11:48

Re: [pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Stephen Clark
Hmm... I don't think pmacct directly support cassandra, but is does support MongoDB. Also I would like to be able to filter/sample the data at the point of origin, but I don't think that is possible with pmacctd and the nfprobe module. It only seems to be able to do it at the collector -

[pmacct-discussion] collecting large number of netflows

2016-08-04 Thread Stephen Clark
Hi List, I am looking to collect a large number of netflow records, on the order of a 100 million a day, and store them in a postgres DB. Has anyone done this or something similar using pmacct? Thanks, Steve ___ pmacct-discussion mailing list