Re: vyatta for bgp
We service most of the state's public schools and libraries (about 1000). Historically the CPE of choice was a small Cisco ISR (1600, 1700, 1800, and 1900 most recently). As bandwidth levels went up, and Ethernet-based transport services became available, we started looking and leveraging FOSS on commodity hardware to lower costs and move services to the edge. Right now we have about 100 of the bigger school districts being services by a Linux-based appliance running XORP for its routing engine (we would have tried Quagga, but they don't support multicast routing yet, nor does Vyatta). It's been a learning experience. Most of the problems we ran into have been resolved by tuning the kernel parameters to act more like a router than a desktop or server. XORP itself has had a rocky ride since we started, so the stability of the project has also been a concern. Thankfully it is seeing somewhat active development again. I will note that XORP is very touchy about how it's configured; if you have well tested configuration templates it's fine, but it's very easy to get it into a crashing state based on something as little the order of configuration directives. For the most part once it's running it's stable. Modest hardware (3.2GHz dual-core Xeon, 2GB RAM, with 1GB tied up as a RAM disk) seems to do the job well for 100 Mbps without much issue, and that's with stateful firewall, and web content filtering in place. Instead of doing it in-house we found a vendor in MA that was doing something similar to what we wanted and had them develop a modified version of their existing offering for us. The vendor is MECnet for those interested. On Thu, Sep 22, 2011 at 6:37 AM, Pierce Lynch p.ly...@netappliant.com wrote: Andreas Echavez [mailto:andr...@livejournalinc.com] originally wrote: Ultimately, the network is as reliable as you build it. With software, it's much cheaper to divide and scale horizontally. Hardware devices are expensive and usually horizontal scalability never happens. So in reality, an enterprise blows 100k on two routers, they both flop because of some firmware bug, and you're down. With this in mind, I am keen to understand how many implementations of packages such as Quagga and Zebra that the group use. With the likes of Vyatta being discussed, I am keen to see if products such as Quagga as still regularly used as it used to be. Thoughts welcome! Kind regards, /P. -- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
Re: vyatta for bgp
--- r...@maine.edu wrote: From: Ray Soucy r...@maine.edu We service most of the state's public schools and libraries (about 1000). Historically the CPE of choice was a small Cisco ISR (1600, 1700, 1800, and 1900 most recently). As bandwidth levels went up, and Ethernet-based transport services became available, we started looking and leveraging FOSS on commodity hardware to lower costs and move services to the edge. Right now we have about 100 of the bigger school districts being services by a Linux-based appliance running XORP for its routing engine (we would have tried Quagga, but they don't support multicast routing yet, nor does Vyatta). It's been a learning experience. Most of the problems we ran into have been resolved by tuning the kernel parameters to act more like a router than a desktop or server. XORP itself has had a rocky ride since we started, so the stability of the project has also been a concern. Thankfully it is seeing somewhat active development again. I will note that XORP is very touchy about how it's configured; if you have well tested configuration templates it's fine, but it's very easy to get it into a crashing state based on something as little the order of configuration directives. For the most part once it's running it's stable. After roll-out and after a time in steady-state operation did you do an analysis of human and hardware/software costs (as well as service to end sites, such as outages that might not have happened with normal routers and LAN switches) to see if you actually saved money? scott
Re: vyatta for bgp
There are a lot of variables that would skew numbers in favor of using FOSS on commodity hardware in our situation, that wouldn't necessarily apply to others. Primarily because these are used to provide services that are in part funded through the federal E-rate program, and need to comply with restrictions such as CIPA. For example, we moved from centralized web filtering using WCCP and racks of proxy servers, to pushing that service out to the edge. That move alone provided more savings than the hardware cost of the project, so we actually made a net profit from the move in our situation. Not sure that would easily apply to anyone. As for the OpEx and CapEx v. traditional players... The units are engineered so that they run the entire OS on a RAM disk; so configuration management is much like what you would find with a traditional router (only saved configuration survives a reboot, etc -- think of it like a live distribution with controlled persistence). A physical disk is used for logging, but does not take out the system upon failure (we've had maybe 3 disk failures that turned out to be thermal conditions of where equipment was installed -- boiler rooms -- and service was maintained until we had a technician out to swap the unit). So operationally, they've been pretty much equivalent of a Cisco solution and we haven't seen much of an increase in activity aside from supporting the extra services that weren't previously available. The skill set is a little different though. Having a strong understanding of the internals of a Linux system along side traditional networking skills is a must if you go in this direction. For us, the ability to have more tools to poke at the state of the system and troubleshoot issues (such as performing packet captures directly on the device) has been invaluable. It has allowed us to track down issues (such as TCP window scaling problems with unnamed cloud services and their incorrectly configured load balancers) remotely that would have required on-site capture in the past. It's also provided us with the flexibility to quickly implement operational changes as we see a need, such as implementing automatic nightly backup of configurations to our central servers (using a simple CRON job), or rolling out scripted changes. Using an off-the-shelf distribution of Linux and a FOSS routing package will probably not do the trick for you. If you take the time to build a custom distribution that only has what you need; makes use of known stable package versions, and is engineered to function as a widely-deployed unit (configuration management, logging, etc) that is where the savings will come in, because you won't need to see the significant increase in OpEx that opponents usually point to. We were debating on if we should do that in-house or not. I think if you're talking about 1000 units then in makes sense to try in-house, on a smaller scale you really want to find a partner that can engineer the system for you. Vyatta looks like it's addressed a lot of the issues it needs to -- though I've never used it in production -- but I would still like to see more from them in tuning the OS to function better as a router and less like a server. Last time I checked they didn't seem to touch much except setting Linux to allow forwarding. I'm optimistic though. Now we just need Intel to step up with some ASICs and open source drivers that could be plugged into Linux. (On a side note, we make use of some SFP PCI-X cards for our direct optical connected sites to save money there too; working well with up to ZX SFPs). On Mon, Sep 26, 2011 at 3:20 PM, Scott Weeks sur...@mauigateway.com wrote: --- r...@maine.edu wrote: From: Ray Soucy r...@maine.edu We service most of the state's public schools and libraries (about 1000). Historically the CPE of choice was a small Cisco ISR (1600, 1700, 1800, and 1900 most recently). As bandwidth levels went up, and Ethernet-based transport services became available, we started looking and leveraging FOSS on commodity hardware to lower costs and move services to the edge. Right now we have about 100 of the bigger school districts being services by a Linux-based appliance running XORP for its routing engine (we would have tried Quagga, but they don't support multicast routing yet, nor does Vyatta). It's been a learning experience. Most of the problems we ran into have been resolved by tuning the kernel parameters to act more like a router than a desktop or server. XORP itself has had a rocky ride since we started, so the stability of the project has also been a concern. Thankfully it is seeing somewhat active development again. I will note that XORP is very touchy about how it's configured; if you have well tested configuration templates it's fine, but it's very easy to get it into a crashing state based on something as little the order of configuration directives. For the most part once it's running it's stable.
Re: vyatta for bgp
On 9/22/11 11:38 , Charles N Wyble wrote: * On 09/22/2011 05:37 AM, Pierce Lynch wrote:** Andreas Echavez [mailto:andreas at livejournalinc.com https://mailman.nanog.org/mailman/listinfo/nanog] originally wrote:** Ultimately, the network is as reliable as you build it. With** software, it's much cheaper to divide and scale horizontally.** Hardware devices are expensive and usually horizontal** scalability never happens. So in reality, an enterprise blows 100k on** two routers, they both flop because of some firmware bug, and** you're down.** With this in mind, I am keen to understand how many implementations of** packages such as Quagga and Zebra that the group use. With the likes** of Vyatta being discussed, I am keen to see if products such as Quagga** as still regularly used as it used to be. I think that the original/upstream versions are out of date as compared** to the one maintained by Vyatta. Or Google (for their MPLS processing** needs). See** http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50** http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50* We are actively supporting Quagga. We currently have a git repo at code.google.com with some BGP multipath updates, and are working with ISC to provide SQA on that branch. Hopefully more features will be forthcoming. Search quagga-dev if you're interested in more details. Vyatta has done a lot of great work on Quagga, as have many others. It would be nice to see all the various useful branches merged into a cherry-picked mainline that would simplify the Quagga development community's lives considerably. -Scott we [opensourcerouting.org (ISC project)] are working on providing SQA around Quagga. Our goal is to enable the community to build a more stable, feature rich version of the quagga baseline.We're providing testing, release management, and helping develop patches, features etc. We have started to test quagga's baseline code (99.18) covering a) compliance (RFCs) and interop (with J and C) b) scenario/functional/scale/performance testing c) resilience and security testing We already found several issues and have started to bug fix with the community. See the quagga-dev list and bugzilla at quagga.net for details. Examples - scale limits on BGP, incorrect route calculation, etc (this is the main branch NOT variants) In addition, we are also testing other branches and benchmarking them. Vyatta's code and google's MP updates are some of many variants we are working with (testing). Over the next few months of testing the mainline, and variants, we will also work with the community (Vyatta, google, independent committers, and others) to facilitate a merged release. We will also test this against different configurations (OSs, Servers, and switches ;p). As part of the merge we will also help review and manage code with the community, leveraging some of the experiences from ISC in bind. If anyone has used Quagga in their network in any sort of configuration, or even modified code to improve it, please contact us (me or i...@opensourcerouting.org). As we are putting together the release and tests for scenario/functional/scale/perf/etc - input would be greatly appreciated. We have a repository also which we can open up for new code/patches etc, but it needs to also be given to the community. As I have stated are working with Vyatta (and google, and others not be mentioned), but more are always welcome. We will be at Nanog in philly - come find me or one of my team members. Thanks Bill
RE: vyatta for bgp
Andreas Echavez [mailto:andr...@livejournalinc.com] originally wrote: Ultimately, the network is as reliable as you build it. With software, it's much cheaper to divide and scale horizontally. Hardware devices are expensive and usually horizontal scalability never happens. So in reality, an enterprise blows 100k on two routers, they both flop because of some firmware bug, and you're down. With this in mind, I am keen to understand how many implementations of packages such as Quagga and Zebra that the group use. With the likes of Vyatta being discussed, I am keen to see if products such as Quagga as still regularly used as it used to be. Thoughts welcome! Kind regards, /P.
Re: vyatta for bgp
On 09/22/2011 05:37 AM, Pierce Lynch wrote: Andreas Echavez [mailto:andr...@livejournalinc.com] originally wrote: Ultimately, the network is as reliable as you build it. With software, it's much cheaper to divide and scale horizontally. Hardware devices are expensive and usually horizontal scalability never happens. So in reality, an enterprise blows 100k on two routers, they both flop because of some firmware bug, and you're down. With this in mind, I am keen to understand how many implementations of packages such as Quagga and Zebra that the group use. With the likes of Vyatta being discussed, I am keen to see if products such as Quagga as still regularly used as it used to be. I think that the original/upstream versions are out of date as compared to the one maintained by Vyatta. Or Google (for their MPLS processing needs). See http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50 http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50 Thoughts welcome! Kind regards, /P.
Re: vyatta for bgp
On 9/22/11 11:38 , Charles N Wyble wrote: On 09/22/2011 05:37 AM, Pierce Lynch wrote: Andreas Echavez [mailto:andr...@livejournalinc.com] originally wrote: Ultimately, the network is as reliable as you build it. With software, it's much cheaper to divide and scale horizontally. Hardware devices are expensive and usually horizontal scalability never happens. So in reality, an enterprise blows 100k on two routers, they both flop because of some firmware bug, and you're down. With this in mind, I am keen to understand how many implementations of packages such as Quagga and Zebra that the group use. With the likes of Vyatta being discussed, I am keen to see if products such as Quagga as still regularly used as it used to be. I think that the original/upstream versions are out of date as compared to the one maintained by Vyatta. Or Google (for their MPLS processing needs). See http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50 http://www.nanog.org/meetings/nanog50/abstracts.php?pt=MTYzNSZuYW5vZzUwnm=nanog50 We are actively supporting Quagga. We currently have a git repo at code.google.com with some BGP multipath updates, and are working with ISC to provide SQA on that branch. Hopefully more features will be forthcoming. Search quagga-dev if you're interested in more details. Vyatta has done a lot of great work on Quagga, as have many others. It would be nice to see all the various useful branches merged into a cherry-picked mainline that would simplify the Quagga development community's lives considerably. -Scott
Re: vyatta for bgp
On Sep 12, 2011, at 11:42, Ben Albee wrote: Does anybody currently use vyatta as a bgp router for their company? If so have you ran into any problems with using that instead of a cisco or juniper router? We're using Vyatta for a handful of fast ethernet links to the internet, with I think about three dozen BGP peers. (Mix of IPv4 and IPv6; about four full feeds on each protocol, the rest is peering). It's not as mature or polished as I understand some of the Cisco or Juniper platforms are; but on our small scale it's fine. We have a decent amount of of Linux expertise in the office (and virtually zero for Juniper/Cisco/...), so having more familiar tools on the routers is nice. As a small shop it's also convenient that the boxes are cheap (so we can have two hot ones with VRRP etc and cheaply a third cold spare) and that the spare parts etc are the same or similar to the rest of the boxes in the rack. - ask -- http://askask.com/
Re: vyatta for bgp
I'll chime in, In an enterprise environment, I've worked with software routers as well as hardware beasts (ala Junipers, Cisco 6500s, ASAs, and more). Ultimately, the network is as reliable as you build it. With software, it's much cheaper to divide and scale horizontally. Hardware devices are expensive and usually horizontal scalability never happens. So in reality, an enterprise blows 100k on two routers, they both flop because of some firmware bug, and you're down. The most reliable/cost effective solution is the cheap and redundant approach to architecture. Reliable hardware is incredibly inexpensive, and every year we get better CPUs and (recently) GPUs that are providing APIs and interfaces to their incredible parallel processing capability. btw, you guys might find PacketShaderhttp://shader.kaist.edu/packetshader/a pretty interesting concept -Andreas On Thu, Sep 15, 2011 at 6:51 AM, Alain Hebert aheb...@pubnix.net wrote: Hi, As usual this end-up in what people prefer. Vyatta is as good as the hardware it runs on, the backend they use and the people configuring/maintaining it. The nature of ASIC make it more reliable than a multi-purpose device (aka server) running an OS written for it. It end up being a choice between risk and cost and being that you can get your hand on second hand iron for cheap these days... Why risk it. - Alain Hebertaheb...@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.netFax: 514-990-9443 On 09/15/11 09:05, Ray Soucy wrote: Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... On Tue, Sep 13, 2011 at 7:36 PM, Dobbins, Rolandrdobb...@arbor.net wrote: On Sep 14, 2011, at 5:54 AM, Deepak Jain wrote: Some enterprises get MPLS L3 VPN service from their providers, and need boxes that can route packets to it and speak BGP to inject their routes. They are not, per se, connected to the Internet, and thus won't be zorched, at least in the sense you are using it. Hence 'public-facing'. ; --**--** --- Roland Dobbinsrdobb...@arbor.net //http://www.arbornetworks.**comhttp://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
On Wed, Sep 21, 2011 at 4:14 PM, Andreas Echavez andr...@livejournalinc.com wrote: The most reliable/cost effective solution is the cheap and redundant approach to architecture. Reliable hardware is incredibly inexpensive, and every year we get better CPUs and (recently) GPUs that are providing APIs and interfaces to their incredible parallel processing capability. -Andreas +1 Scaling Horizontally. Applies to your networking gear, your applications, etc. If you assume anything is going to break, just get more and scale/architect properly. On Thu, Sep 15, 2011 at 6:51 AM, Alain Hebert aheb...@pubnix.net wrote: Hi, As usual this end-up in what people prefer. Vyatta is as good as the hardware it runs on, the backend they use and the people configuring/maintaining it. The nature of ASIC make it more reliable than a multi-purpose device (aka server) running an OS written for it. It end up being a choice between risk and cost and being that you can get your hand on second hand iron for cheap these days... Why risk it. - Alain Hebertaheb...@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.netFax: 514-990-9443 On 09/15/11 09:05, Ray Soucy wrote: Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... On Tue, Sep 13, 2011 at 7:36 PM, Dobbins, Rolandrdobb...@arbor.net wrote: On Sep 14, 2011, at 5:54 AM, Deepak Jain wrote: Some enterprises get MPLS L3 VPN service from their providers, and need boxes that can route packets to it and speak BGP to inject their routes. They are not, per se, connected to the Internet, and thus won't be zorched, at least in the sense you are using it. Hence 'public-facing'. ; --**--** --- Roland Dobbinsrdobb...@arbor.net //http://www.arbornetworks.**com http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde -- Brandon Galbraith US Voice: 630.492.0464
Re: vyatta for bgp
On 09/21/2011 06:14 PM, Andreas Echavez wrote: btw, you guys might find PacketShaderhttp://shader.kaist.edu/packetshader/a pretty interesting concept -Andreas Excellent! I was wondering how far along this was. Good to see. Very exciting. I've got a couple parallel systems sitting around looking for packets to route... If anyone is doing research in this area, please let me know. Most of my research has been into accelerating IDS/IPS and fuzzing workloads with parallel systems. (Yes that's on top of starting an ISP). I've been looking into http://www.read.cs.ucla.edu/click/Click
Re: vyatta for bgp
Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... On Tue, Sep 13, 2011 at 7:36 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Sep 14, 2011, at 5:54 AM, Deepak Jain wrote: Some enterprises get MPLS L3 VPN service from their providers, and need boxes that can route packets to it and speak BGP to inject their routes. They are not, per se, connected to the Internet, and thus won't be zorched, at least in the sense you are using it. Hence 'public-facing'. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde -- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
Re: vyatta for bgp
Hi, As usual this end-up in what people prefer. Vyatta is as good as the hardware it runs on, the backend they use and the people configuring/maintaining it. The nature of ASIC make it more reliable than a multi-purpose device (aka server) running an OS written for it. It end up being a choice between risk and cost and being that you can get your hand on second hand iron for cheap these days... Why risk it. - Alain Hebertaheb...@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.netFax: 514-990-9443 On 09/15/11 09:05, Ray Soucy wrote: Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... On Tue, Sep 13, 2011 at 7:36 PM, Dobbins, Rolandrdobb...@arbor.net wrote: On Sep 14, 2011, at 5:54 AM, Deepak Jain wrote: Some enterprises get MPLS L3 VPN service from their providers, and need boxes that can route packets to it and speak BGP to inject their routes. They are not, per se, connected to the Internet, and thus won't be zorched, at least in the sense you are using it. Hence 'public-facing'. ; --- Roland Dobbinsrdobb...@arbor.net //http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
Ray Download the Podcast The Packet Pushers - Show 31 they talk a little about this topic... If nothing else it's a great listen Cheers! On Thu, Sep 15, 2011 at 11:05 PM, Ray Soucy r...@maine.edu wrote: Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... -- Regards, Jason Leschnik. Mob. 0432 35 4224 Uni mail. jml...@uow.edu.au
Re: vyatta for bgp
Thanks for the tip, first time I hear this podcast. On Thu, Sep 15, 2011 at 9:58 AM, Jason Leschnik lesch...@gmail.com wrote: Ray Download the Podcast The Packet Pushers - Show 31 they talk a little about this topic... If nothing else it's a great listen Cheers! On Thu, Sep 15, 2011 at 11:05 PM, Ray Soucy r...@maine.edu wrote: Is Vyatta really not suited for the task? I keep checking up on it and holding off looking into it as they don't support multicast yet. Modern commodity sever hardware these days often out-powers big iron enough to make up for not using ASICs, though, at least on the lower end of the spectrum. Does anyone have any more details on Vyatta not scaling? Were you trying to run it as a VM? What were you using for NICs? etc. The hardware matters. Saying Vyatta doesn't cut it could mean anything... -- Regards, Jason Leschnik. Mob. 0432 35 4224 Uni mail. jml...@uow.edu.au -- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
Re: vyatta for bgp
On Mon, 2011-09-12 at 15:41 -0400, Jared Geiger wrote: There was a bug where you couldn't use two IPv4 peers and then add IPv6. I haven't tested the newest versions yet to see if it still exists. Works great for two IPv4 peers. Discussion between developers on bugfixes can often be seen in ##vyatta on Freenode. :) I find it interesting to idle/chime-in occasionally at least. Tom
Re: vyatta for bgp
On Mon, 12 Sep 2011 20:48:31 CDT, Jimmy Hess said: One thing.. the OP was asking about anyone using Vyatta for BGP. Using Vyatta for BGP doesn't necessarily mean the Vyatta unit is actually a device forwarding the packets... someone could be using it as a route server, or for otherwise populating forwarding tables of other devices with third-party next hops :-) I would expect a properly configured Vyatta running as a route server to be pretty darn near zortch-proof, no? (Barring BGP packet-o-death issues of course - but is there a router vendor who *hasn't* had at least 2 or 3 of those? ;) pgpSmyxnBmYIO.pgp Description: PGP signature
Re: vyatta for bgp
On Mon, 12 Sep 2011 22:38:57 BST, Nick Hilliard said: Let's throw some figures around (ridiculously simplified): a company has a choice between a pair of $10k software routers or something like a pair of MX80s for $25k each. So, one solution costs $20k; the other $50k. $30k cost difference works out as $625 per month depreciation (4 year). I.e. not going to affect the bottom line in any meaningful way. Now say that this company has a DoS attack for 24h, and the company effectively loses one day of revenue. On the basis that there are 260 office working days per year, the point at which spending an extra $30k for a hardware router would be of net benefit to the company would be 260*30k = $7.8m. I.e. if your annual revenue is higher than that, and if spending that cash would mitigate against your DoS problems, then it would be worth your while in terms of direct loss mitigation. Of course, this analysis is quite simplistic and excludes things like damage to reputation, online stores, the likelihood of DoS attacks happening in the first place, the cost of transit and many other points of reality. One important thing it overlooks is what percent of DDoS attackqs are simple flood the pipe attacks directed at a target behind the router. If you got a 100M or 1G pipe to the outside world and you're getting hammered by multiple G worth of packets, things are going to suck no matter what the router is. And let's face it, kicking that pipe to 10G is gonna cost a bit pgpaOKFyCFolN.pgp Description: PGP signature
Re: vyatta for bgp
On Sep 13, 2011, at 1:42 AM, Ben Albee wrote: Does anybody currently use vyatta as a bgp router for their company? The days of public-facing software-based routers were over years ago - you need an ASIC-based edge router, else you'll end up getting zorched. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
The days of public-facing software-based routers were over years ago - you need an ASIC-based edge router, else you'll end up getting zorched. wait, what? -- //fredan
RE: vyatta for bgp
-Original Message- From: Dobbins, Roland [mailto:rdobb...@arbor.net] Sent: Monday, September 12, 2011 11:56 AM To: North American Network Operators' Group Subject: Re: vyatta for bgp On Sep 13, 2011, at 1:42 AM, Ben Albee wrote: Does anybody currently use vyatta as a bgp router for their company? The days of public-facing software-based routers were over years ago - you need an ASIC-based edge router, else you'll end up getting zorched. How do you come to this conclusion? I think a software-based router for enterprise level (let's say on the 1G per provider level) can handle a fair amount of zorching. I checked the Cisco and Juniper docs and neither vendor is anywhere near releasing their anit-zorching ASICs. Mike
Re: vyatta for bgp
On 12/09/2011 20:08, Michael K. Smith - Adhost wrote: How do you come to this conclusion? I think a software-based router for enterprise level (let's say on the 1G per provider level) can handle a fair amount of zorching. I presume by a fair amount, I presume you mean barely any? At large packet sizes, an enterprise level router will just about handle a 1G DoS attack. Thing is, bandwidth DoS / DDoS is sufficiently easy to pull off on a large scale that a 1G DoS is pretty easy. Incidentally, most service providers use enterprise level as a by-word for mediocre quality kit, lacking in both stability and useful features. Nick
Re: vyatta for bgp
On Sep 12, 2011, at 12:35 PM, Nick Hilliard wrote: On 12/09/2011 20:08, Michael K. Smith - Adhost wrote: How do you come to this conclusion? I think a software-based router for enterprise level (let's say on the 1G per provider level) can handle a fair amount of zorching. I presume by a fair amount, I presume you mean barely any? At large packet sizes, an enterprise level router will just about handle a 1G DoS attack. Thing is, bandwidth DoS / DDoS is sufficiently easy to pull off on a large scale that a 1G DoS is pretty easy. Incidentally, most service providers use enterprise level as a by-word for mediocre quality kit, lacking in both stability and useful features. Nick In your typical enterprise environment, a 1G DoS will zorch the link long before it zorches the router at the enterprise side. I agree that software-based routers are not a good choice for a backbone provider, but, for an enterprise that is dealing with 1gbps links coming in from ≤3 providers, the difference in cost makes a software router an attractive option in many cases. Of course it is important to understand the limitations of the solution you choose, but, in such an environment, a USD100,000+ ASIC based router may be like trying to kill a mosquito with a sledge hammer. Owen smime.p7s Description: S/MIME cryptographic signature
Re: vyatta for bgp
On Sep 13, 2011, at 2:45 AM, Owen DeLong wrote: In your typical enterprise environment, a 1G DoS will zorch the link long before it zorches the router at the enterprise side. This contradicts my experience - I've repeatedly witnessed only a few mb/sec of 64-byte packets making software-based routers fall over, including just last month. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
RE: vyatta for bgp
Original Message- From: Dobbins, Roland [mailto:rdobb...@arbor.net] Sent: Monday, September 12, 2011 2:56 PM To: North American Network Operators' Group Subject: Re: vyatta for bgp zorched. --- Zorch. I like that. Sounds like a Batman fight-scene bubble word. Is the concern over a DDOS aimed against the router itself, or just massive flows passing through? Chuck
Re: vyatta for bgp
On Mon, 12 Sep 2011 20:12:43 -, Dobbins, Roland said: This contradicts my experience - I've repeatedly witnessed only a few mb/sec of 64-byte packets making software-based routers fall over, including just last month. On the flip side, there's a *lot* of sites that have to make trade-offs, and the risk that their $10K software-based router may fall over doesn't justify adding another zero to the price tag, especially if their network includes a lot of branch offices that would all add another zero pgpzyUSU71dRV.pgp Description: PGP signature
Re: vyatta for bgp
On Sep 13, 2011, at 3:34 AM, Chuck Church wrote: Is the concern over a DDOS aimed against the router itself, or just massive flows passing through? Yes, but mainly the former. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
Thanks for the all the feed-back. We will only have two ipv4 BGP peers (both 5mb/sec links) to the same ISP. We are doing BGP because we plan to add a second ISP at one of our locations in the future. We are not any near a large enterprise, this will be replacing two DSL lines and a T1.
Re: vyatta for bgp
On Sep 13, 2011, at 3:43 AM, Everton Marques wrote: Would Cisco ISR G2 3925E classify as software-based router? Yes. Do you expect it to bend itself down under a few Mbps of 64-byte packets? Especially if they're directed at the router itself, at some point, sure - though the ISR2 certainly has more horsepower than the original ISRs, and I've personally yet to witness an ISR2 being DDoSed, so I've no feel for the specific numbers. Features also play a role. This isn't to say that the ISR2 isn't a fine router - but rather that one must be cognizant of performance envelopes prior to deployment in order to determine suitability to purpose. One can't reasonably expect vendors to exceed their design constraints in any type of equipment. ; One can and should test the specific performance envelope of any prospective infrastructure purchase, of course. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
On Mon, Sep 12, 2011 at 1:52 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Sep 13, 2011, at 3:43 AM, Everton Marques wrote: Would Cisco ISR G2 3925E classify as software-based router? Yes. Do you expect it to bend itself down under a few Mbps of 64-byte packets? Especially if they're directed at the router itself, at some point, sure - though the ISR2 certainly has more horsepower than the original ISRs, and I've personally yet to witness an ISR2 being DDoSed, so I've no feel for the specific numbers. Features also play a role. This isn't to say that the ISR2 isn't a fine router - but rather that one must be cognizant of performance envelopes prior to deployment in order to determine suitability to purpose. One can't reasonably expect vendors to exceed their design constraints in any type of equipment. ; One can and should test the specific performance envelope of any prospective infrastructure purchase, of course. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde Lots of devices can have trouble if you direct high PPS to the control plane, and will exhibit performance degradation, leading up to a DoS eventually. That isn't limited to software based routers at all, it will impact dedicated ASICs. Vendors put together solutions for this, to protect the router itself/control plane, whether its a software based routed or ASICs. Now if this was a Microtik with an 1Ghz Intel Atom CPU, sure, lots of things could take that thing offline, even funny looks. But a modern, multi-core/multi-thread system with multi-queued NICs will handle hundreds of thousands of PPS directed to the router itself before having issues, of nearly any packet size. A high end ASIC can handle millions/tens of millions PPS, but directed to the control plane (which is often a general purpose CPU as well, Intel or PowerPC), probably not in most scenarios. I think its very fair for a small/medium sized organization to run software based routers, Vyatta included. -- Brent Jones br...@servuhome.net
Re: vyatta for bgp
On Sep 13, 2011, at 4:13 AM, Brent Jones wrote: A high end ASIC can handle millions/tens of millions PPS, but directed to the control plane (which is often a general purpose CPU as well, Intel or PowerPC), probably not in most scenarios. CoPP. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde
Re: vyatta for bgp
Brent, On Mon, Sep 12, 2011 at 11:13 PM, Brent Jones br...@servuhome.net wrote: Lots of devices can have trouble if you direct high PPS to the control plane, and will exhibit performance degradation, leading up to a DoS eventually. That isn't limited to software based routers at all, it will impact dedicated ASICs. Vendors put together solutions for this, to protect the router itself/control plane, whether its a software based routed or ASICs. Now if this was a Microtik with an 1Ghz Intel Atom CPU, sure, lots of things could take that thing offline, even funny looks. But a modern, multi-core/multi-thread system with multi-queued NICs will handle hundreds of thousands of PPS directed to the router itself before having issues, of nearly any packet size. A high end ASIC can handle millions/tens of millions PPS, but directed to the control plane (which is often a general purpose CPU as well, Intel or PowerPC), probably not in most scenarios. I think its very fair for a small/medium sized organization to run software based routers, Vyatta included. Speaking of Mikrotik there, I recently pushed 350kpps small packets through an x86 routeros image running under kvm (using vt-d for nic) on my desktop machine (which is a number i seem to run into more than once when it comes to linux/linux-derivative forwarding on single queue core). I saw a release note claiming their next sw release will do 15-20% more on both mips and x86. Unsurprisingly is open source software forwarding very far from 10G linerate of small pps through single cpu core still. 350kpps of 64B packets is of course merely 180 Mbps (notably, actually sufficient for handling incoming small packets on a 100 Mbps uplink). Re adversaries or random scum filling your uplinks with useless bits, I think I hear the largest DDoS'es now have filled 100G links, so.. don't make yourself a packeting target if you happen to run smaller links than that? :) Generally on staying alive through DDoS by anything else than some degree of luck, I guess having more bandwith between your network and your peers than what your peers all have to their peers is advised (the statement could possibly be improved upon using some minimum cut graph theory language). Best, Martin
Re: vyatta for bgp
On Mon, Sep 12, 2011 at 2:35 PM, Nick Hilliard n...@foobar.org wrote: I presume by a fair amount, I presume you mean barely any? At large packet sizes, an enterprise level router will just about handle a 1G DoS attack. Thing is, bandwidth DoS / DDoS is sufficiently easy to [snip] How much zorching a software router can take depends on a lot of factors. If the hardware necessary to size appropriately for the link is economical and sufficient, zorching is not the largest concern. 1G link speed and 100M link speed offer very different worst-case scenarios; the link can be zorched long before the router is. A software router running in a 32bit OS on an old Pentium 4 can take a lot less zorching than a router running on a server with 6-core 4Ghz CPUs, when interrupt coalescing is present and utilized efficiently. Hardware basic routers have a lower forwarding latency, which makes them more suitable for ISP/carrier networks, the hop delay penalty is lower, and jitter might be a concern on a router running a non real-time OS such as a vanilla Linux kernel or other OS not specially designed for the router task, but there's otherwise nothing wrong with appropriately specc'ed software forwarders. One thing.. the OP was asking about anyone using Vyatta for BGP. Using Vyatta for BGP doesn't necessarily mean the Vyatta unit is actually a device forwarding the packets... someone could be using it as a route server, or for otherwise populating forwarding tables of other devices with third-party next hops :-) -- -JH
Re: vyatta for bgp
On 9/12/2011 3:12 PM, Dobbins, Roland wrote: On Sep 13, 2011, at 2:45 AM, Owen DeLong wrote: In your typical enterprise environment, a 1G DoS will zorch the link long before it zorches the router at the enterprise side. This contradicts my experience - I've repeatedly witnessed only a few mb/sec of 64-byte packets making software-based routers fall over, including just last month. --- Roland Dobbinsrdobb...@arbor.net //http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde +1 tv
