Re: Service Provider NetFlow Collectors

rg Cc: nanog@nanog.org Subject: Re: Service Provider NetFlow Collectors Erik, Feel free to ping me, I own Mimir Networks, we have a full-service flow collection/DDoS detection and mitigation system that I'd love to show you. We built it having been a long time user of other commercial and open s

Re: Service Provider NetFlow Collectors

Erik, Feel free to ping me, I own Mimir Networks, we have a full-service flow collection/DDoS detection and mitigation system that I'd love to show you. We built it having been a long time user of other commercial and open source tools, for very large deployments. Would be happy to give you a

Re: Service Provider NetFlow Collectors

t;Erik Sundberg" mailto:esundb...@nitelusa.com>> > Cc: nanog@nanog.org<mailto:nanog@nanog.org> > Sent: Monday, December 31, 2018 3:40:40 AM > Subject: Re: Service Provider NetFlow Collectors > > Don’t underestimate good old ELK > https://www.elastic.co/guide/en/lo

Re: Service Provider NetFlow Collectors

Throwing my hat in the ring also (vendor from fmadio) https://github.com/fmadio/pcap2json Not exactly a newflow collector, its pcap -> flowgen -> elk on a single box, working very well so far, still work in progress. Problem with logstash is its too slow for high flow rates. So we did everything

Re: Service Provider NetFlow Collectors

erg" mailto:esundb...@nitelusa.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Monday, December 31, 2018 3:40:40 AM Subject: Re: Service Provider NetFlow Collectors Don’t underestimate good old ELK https://www.elastic.co/guide/en/logstash/current/netflow-module.html + https:/

Re: Service Provider NetFlow Collectors

Hi Saku, aggregate [DSTAS]: label, dst_as, peer_dst_as, out_iface aggregate [SRCAS]: label, src_as, peer_src_as, in_iface aggregate[IP]: label, dst_as, src_host, out_iface, in_iface And a script goes over this output to relate ifindex to ifalias from also influxdb SNMP counter DB (where the

Re: Service Provider NetFlow Collectors

Hey Phil, What use cases are you trying to work on? I work for Kentik on the product side of things and this kind of info is very interesting for me to hear. Happy to take your reply in a DM or here. Dan On Wed, Jan 2, 2019 at 6:01 AM Phil Lavin wrote: > > Doesn't Kentik cost like $2000 a

RE: Service Provider NetFlow Collectors

> Doesn't Kentik cost like $2000 a month minimum? We recently got a quote from Kentik and I fell off my chair. The annual cost was slightly more than the total upfront purchase cost of the hardware they were collecting Flow from and was significantly more than the total cost each year of

Re: Service Provider NetFlow Collectors

That’s a much better cardinality (AS based) but it’s not the general case. Even if you want per-prefix information I’d argue that Influx would still not handle the load (~700k ^ 2 cardinality). For limited tag-sets it would do the trick. I never did attempt to push it to Influx with some

Re: Service Provider NetFlow Collectors

Hey, On Wed, 2 Jan 2019 at 14:40, H I Baysal wrote: > That absolutely depends on the amount of TAGs you use, and how you aggregate, > etc. > I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even > sweating a single drop > > We have a 4 Tbps of traffic during peak, and as

Re: Service Provider NetFlow Collectors

This is correct, With a flow database you want to be able to say: “show me all HTTP traffic from subnet a.b.c.0/24” which requires you to either keep individual IPs or aggregate subnets. Combined with port and protocol data for both source and destination, the series count shoots way above

Re: Service Provider NetFlow Collectors

Hi Tim, That absolutely depends on the amount of TAGs you use, and how you aggregate, etc. I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even sweating a single drop We have a 4 Tbps of traffic during peak, and as well as pmacct and influxdb or running very very

Re: Service Provider NetFlow Collectors

Hey Tim, > I would advise against InfluxDB in this case - flow data has a very high (and > open) tag cardinality which is not suited to Influx (although their recently > new index format has improved this). I'm not entirely sure I understand. Does this mean the permutations of tags are high,

Re: Service Provider NetFlow Collectors

I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not suited to Influx (although their recently new index format has improved this). I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar store) with a summing

Re: Service Provider NetFlow Collectors

PMACCT (Works Awesome) push to influxdb ( Works awesome) With some custom scripts to add/match interface descriptions. And you can query whatever you want in grafana :D And grafana has a nice API for rendering a dashboardgraph to a PNG and you can send this png to whatever chat/bot or mail you

Re: Service Provider NetFlow Collectors

We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded like Erik might be looking for). Feel free to ping me if anyone

Re: Service Provider NetFlow Collectors

Doesn't Kentik cost like $2000 a month minimum? On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker wrote: > +1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to > worry about it. > > On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" < > nanog-boun...@nanog.org on

Re: Service Provider NetFlow Collectors

+1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to worry about it. On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" wrote: +1 Kentik ... We've been using their DDoS/RTBH mitigation with good success. On 12/31/18 3:52 AM, Eric

RE: Service Provider NetFlow Collectors

On Behalf Of Erik Sundberg Sent: Sunday, December 30, 2018 10:29 PM To: nanog@nanog.org Subject: Service Provider NetFlow Collectors Hi Nanog We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect netflow data off their Core and Edge Routers

Re: Service Provider NetFlow Collectors

anog@nanog.org Sent: Monday, December 31, 2018 3:40:40 AM Subject: Re: Service Provider NetFlow Collectors Don’t underestimate good old ELK https://www.elastic.co/guide/en/logstash/current/netflow-module.html + https://github.com/robcowart/elastiflow BR, ic On 31 Dec 2018, at 04:29, Er

Re: Service Provider NetFlow Collectors

+1 Kentik ... We've been using their DDoS/RTBH mitigation with good success. On 12/31/18 3:52 AM, Eric Lindsjö wrote: Hi, We use kentik and we're very happy. Works great, tons of new features coming along all the time. Going to start looking into ddos detection and mitigation soon. Would

Re: Service Provider NetFlow Collectors

An other tool worth looking into is Traffic Sentinel from inMon. Karsten Am Mo., 31. Dez. 2018 um 04:31 Uhr schrieb Erik Sundberg : > > Hi Nanog…. > > > > We are looking at replacing our Netflow collector. I am wonder what other > service providers are using to collect netflow data off their

Re: Service Provider NetFlow Collectors

Hi, I am always peeking at this OSS project for new installations https://github.com/VerizonDigital/vflow - but did not try it out myself so far. Jörg On 31 Dec 2018, at 4:29, Erik Sundberg wrote: Hi Nanog We are looking at replacing our Netflow collector. I am wonder what other

Re: Service Provider NetFlow Collectors

Hi, We use kentik and we're very happy. Works great, tons of new features coming along all the time. Going to start looking into ddos detection and mitigation soon. Would recommend. Kind regards, Eric Lindsjö On 12/31/2018 04:29 AM, Erik Sundberg wrote: Hi Nanog…. We are looking at

Re: Service Provider NetFlow Collectors

Don’t underestimate good old ELK https://www.elastic.co/guide/en/logstash/current/netflow-module.html + https://github.com/robcowart/elastiflow BR, ic > On 31 Dec 2018, at 04:29,

Re: Service Provider NetFlow Collectors

I’m still using nfsen/nfdump Been looking at manageengine netflow analyzer lately and liking it, we might be buying some time on Calix flowanalyze which might be an improved version of xangati Aaron > On Dec 30, 2018, at 10:44 PM, Michael Gehrmann > wrote: > > > Add Flowtraq to your

Re: Service Provider NetFlow Collectors

Add Flowtraq to your list. Cheers Mike On Mon, 31 Dec 2018 at 14:30, Erik Sundberg wrote: > Hi Nanog…. > > > > We are looking at replacing our Netflow collector. I am wonder what other > service providers are using to collect netflow data off their Core and Edge > Routers. Pros/Cons… What to

Service Provider NetFlow Collectors

Hi Nanog We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect netflow data off their Core and Edge Routers. Pros/Cons... What to watch out for any info would help. We are mainly looking to analyze the netflow data. Bonus if it