Re: Zebra/linux device production networking?
On Tue, Jun 06, 2006 at 02:42:36PM -0700, Nick Burke [EMAIL PROTECTED] wrote a message of 39 lines which said: How many of you have actually use(d) Zebra/Linux as a routing device IMHO, the question is not perfectly phrased. You actually have several issues: * use a regular PC instead of big and expensive iron, * use Linux instead of FreeBSD or IOS or JunOS, * use Zebra instead of Quagga or Xorp. These questions are partly independent and should be addressed as such. For instance, Quagga + a free Unix can run on dedicated boxes like the Soekris, who have different characteristics than a regular PC (no moving parts, for instance). One last advice: be very careful when you read claims like it may seem appealing to suits with no networking knowledge: many people never tried what they criticize, they just do not want their CEO to discover that the expensive network could have been done for much less. [I installed, in a former job, Debian + Linux + Zebra on PCs and they route fine.]
Re: Zebra/linux device production networking?
First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. It is primarily small companies that use zebra or Quagga or openbgpd or Xorp or the Click Modular Router project. There is more than one choice so do your research. The main drawback of all of these is that you cannot get PCI-bus cards that support some common circuit types and the PCI bus cannot handle switching high traffic volumes. Many people build and sell routers based on a PC server running UNIX. They work fine if they are no stretched beyond the role intended. Cisco routers are the same. Look at the limitations of the 2500/2600 series for instance. Some URLs of interest: http://www.read.cs.ucla.edu/click/ http://www.xorp.org/ http://www.openbgpd.org/ http://www.quagga.net/ http://www.zebra.org/ Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. This tends to be a list focused on the cult of the BIG IRON, namely Cisco and Juniper. The people who use PC-based routers have their own hangouts. My main piece of advice is to seek out those hangouts and ask your questions there. Here's the article that started it all (this was featured on /., so likely you've read it already). Sorry, haven't seen these. --Michael Dillon
RE: Zebra/linux device production networking?
I would be interested to know how many software (for want of a better description) routers are in live production in this kind of environment i.e. the 99.% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. It is indeed very common. That is why there are several implementations of BGP and routing software available. These are used in dozens and dozens of commercial products some of which are sold as IP routers, plain and simple. In any case, 5 nines and 6 nines are not always what the marketing department claims. They often exclude planned maintenance periods so if you reboot once a week or you have a crash after changing a config, that doesn't count against the 5 nines. In addition, the 5 nines figure generally applies to the network, not to individual devices within it. Networks can be designed so that the failure of a device does not cause a network outage. This whole issue is so complex that you just can't make blanket recommendations. Even the biggest networks don't just buy and deploy big iron. They run every new router model and software release through an extensive battery of tests. Then they write operational guidelines telling people which features can be used in which situations. They do this to avoid crashes and network outages because the big iron (Cisco/Juniper) simply cannot provide that on its own. A smart small company can get excellent results from Linux routers (although I would take a serious look at FreeBSD or OpenBSD for this). Process is as important as hardware. --Michael Dillon
Re: Zebra/linux device production networking?
Nick Burke wrote: Greetings fellow nanogers, How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? Just have a look for MTU. If you connect home - aDSL - someplace and your MTU is smaller than the aDSL packetsize then your connection is home - adsl - tunnel - someplace That tunnel consists of two routers, linux or whatever. Behind the tunnel you might find some 200 hosts. The speed is 2Meg through the tunnel. It used to connect one /18 and a handful of /24 The two linux boxes were maintained by a guru. They almost never gave problems. Mostly the hardware router behind that tunnel did. I dont know what kind of device it is. All I know is, it seems to know some 8 or more interfaces, hardware or virtual. The installation, a nuclear bunker, used to house some websites and services. (And an XTC-lab :) There are a lot of network bunkers arround. I guess half of them looks the same. Cheers Peter and Karin Dambier -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Graeffstrasse 14 D-64646 Heppenheim +49(6252)671-788 (Telekom) +49(179)108-3978 (O2 Genion) +49(6252)750-308 (VoIP: sipgate.de) mail: [EMAIL PROTECTED] mail: [EMAIL PROTECTED] http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/
Re: Zebra/linux device production networking?
On Wed, 7 Jun 2006 [EMAIL PROTECTED] wrote: First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. It is primarily small companies that use zebra or Quagga or openbgpd or Xorp or the Click Modular Router project. There is more than one choice so do your research. The main drawback of all of these is that you cannot get PCI-bus cards that support some common circuit types and the PCI bus cannot handle switching high traffic volumes. I've talked to people using PC-based system on OC48 and analyzing that entire data. Sounded unbelievable to me but their numbers of how much data PCI(Express) can handle support that PC-based router would be able to do it. How reliable this is and if cost of supporting such router is worth going forward is another matter. Also both Linux and Freebsd are fairly equivalent as bases for such routers and if you have knowledgeable people (and you should if you're considering going with PC router), you should be able to set linux that is secure as freebsd. There are some differences in the routing code whereas Linux is designed with per-flow based switching in mind (which works very well when used as a server) and has extensive packet classification mechanism (which I strongly advise you test in the lab before trying in production). Freebsd has what I consider to be simpler code design for which many believe works better if you receive unusual packets, but personally I've used Linux as packet firewall at Gb rate and it handled DoS fine. Linux also supports multiple routing tables in the kernel, which I think latest quagga can take advantage of and it can make a difference whe selecting linux vs freebsd. Now do remember that biggest headache is going to be supporting this as such custom solution will require custom coding of tools and good engineer who really knows well both linux and networking and finding more such people to support your infrastructure if you grow maybe difficult. -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: Zebra/linux device production networking?
On 6/7/06, Peter Dambier [EMAIL PROTECTED] wrote: The installation, a nuclear bunker, used to house some websites and services. (And an XTC-lab :) Ah, I sometimes wonder about how people get the idea of deploying alternate roots. Then I see that email from Peter and it all becomes blindingly clear. :) --srs -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: Zebra/linux device production networking?
On 6/7/06, Nick Burke [EMAIL PROTECTED] wrote: First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. This looks reasonable .. http://www.linux-vpn.de/lr101.php -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: Zebra/linux device production networking? (summary)
Thanks to all for all the feedback! It seems what a lot of people are saying is that it's almost acceptable (in that, you shouldn't if you can afford other devices), given the right time and engineering. The cost of supporting seems to be unanimously higher then going with a specific vendor. A number of people have noted that some of the support that the various packages of software for handling routing protocols may not play correctly with the os layer or even other packages. (IE: routing) I've seen confliction on if *bsd or linux is better, this (hopefully) isn't that surprising to anyone. The consensus is that when something breaks it takes longer to fix and requires greater technical aptitude. Finally, it appears as if, contrary to what the articles are saying, not many people are actively considering such a move. However, it is more common in smaller businesses starting new locations or building out. A lot of people seemed to of assumed the absolute worse case (which, might I add, is generally what I was looking for) scenario: a dusty box with interesting hardware out-of-the-box kernel no research a MSMD approach What about better case situations?* IE: toe cards custom kernel no moving parts (ie: hard drive, maybe fans if possible) up-to-date software packages with internal coders to fix ugly bugs, etc actual research into what packages hardware would be best *This deviates from operational and gets into the more technical issues, so it's actually a not a question I'm looking for you kind folks to answer. But I feel I have to vindicate myself a little bit as my technical skills were called into question for even posting the original email... ;) Once again, thanks everyone!
Re: Zebra/linux device production networking? (summary)
On Wed, 7 Jun 2006, Nick Burke wrote: What about better case situations?* IE: toe cards custom kernel no moving parts (ie: hard drive, maybe fans if possible) up-to-date software packages with internal coders to fix ugly bugs, etc actual research into what packages hardware would be best I didn't notice anyone mention Imagestream, who sell Linux based routers using a custom distro and no moving parts other than fans. Storage is flash. I've helped a client manage several of them for several years. IMO, they're not bad as CPE, but I don't think we could use them if we wanted to on most of our network. Some of the features we need just aren't available. As others have mentioned, I wouldn't recommend it unless you have some people very comfortable with Linux and IP routing on Linux on staff. At one point, they had 4 full BGP feeds going into one Imagestream Gateway router, which is a P4, upgraded to 512MB RAM. With 2 full views now, they have 308MB free. It's an older installation, predating the addition of zebra/quagga to their distro, so it's still running gated_public, which works, but is fairly lacking in BGP knobs. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: Zebra/linux device production networking?
In article [EMAIL PROTECTED], william(at)elan.net [EMAIL PROTECTED] wrote: you should be able to set linux that is secure as freebsd. There are some differences in the routing code whereas Linux is designed with per-flow based switching in mind (which works very well when used as a server) Nobody noticed, but Linux 2.6 has alternative FIB code you can select when compiling the kernel. Yes, it is fairly new and I'm not sure it is production quality, but still. The config option is IP_FIB_TRIE, for the LC-trie algorithm. It's supposed to be something like CEF. Mike.
Re: Zebra/linux device production networking? (summary)
I've seen confliction on if *bsd or linux is better, this (hopefully) isn't that surprising to anyone. You should do a PPS throughput analysis of your own to see which OS works better on the hardware that you plan to use. Drivers, and the susceptibility of the kernel to livelock, are where there may be differences in performance. Finally, it appears as if, contrary to what the articles are saying, not many people are actively considering such a move. However, it is more common in smaller businesses starting new locations or building out. DEC's gateway to the Internet ran on host-based routers - DEC Alphas running Digital UNIX with turbochannel FDDI cards - from 1994 to sometime in 1999-ish (I stopped being responsible for it in 1998). I started with a pair and had suffered one all-night upgrade to eight when the PPS load of some AltaVista announcement pushed the pair over the edge into livelock. What about better case situations?* IE: toe cards TOE won't help you, you aren't terminating TCP sessions on the box. At least you shouldn't be. Don't let anyone talk you into also running a web server. custom kernel This could be useful, if the kernel is able to handle all packet forwarding in the interrupt or polling input service routine. no moving parts (ie: hard drive, maybe fans if possible) That'll certainly help with reliability, as well as dual power supplies. up-to-date software packages with internal coders to fix ugly bugs, etc actual research into what packages hardware would be best Both of those things, or a support agreement from one of the vendors that's trying to make the host-based open-source router business model work. Stephen
Re: Zebra/linux device production networking?
I'm running ImageStream routers for the Internet distribution side of my network (2 edge routers, 2 core routers) and I'm extremely happy... This is a datacenter network and my customers are happy, I guess that's all that counts. In my opinion, I prefer to go with a open-source based solution because of pricing and customizability... I can build a script and load it into the equipment to give me any type of statistic I want... And I don't have to wait for a new IOS release. JP On 6/7/06, Miquel van Smoorenburg [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], william(at)elan.net [EMAIL PROTECTED] wrote: you should be able to set linux that is secure as freebsd. There are some differences in the routing code whereas Linux is designed with per-flow based switching in mind (which works very well when used as a server) Nobody noticed, but Linux 2.6 has alternative FIB code you can select when compiling the kernel. Yes, it is fairly new and I'm not sure it is production quality, but still. The config option is IP_FIB_TRIE, for the LC-trie algorithm. It's supposed to be something like CEF. Mike. -- Justin W. Pauler Baton Rouge, LA
Re: Zebra/linux device production networking?
On Wed, Jun 07, 2006 at 09:31:51PM +0530, Suresh Ramasubramanian wrote: On 6/7/06, Nick Burke [EMAIL PROTECTED] wrote: First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. This looks reasonable .. http://www.linux-vpn.de/lr101.php LEAF http://leaf.sourceforge.net/ and Coyote http://www.coyotelinux.com/ are often cited live branches off the Linux Router Project. -- Joe Yao --- This message is not an official statement of OSIS Center policies.
Re: Zebra/linux device production networking?
On Wed, 7 Jun 2006, Justin W. Pauler wrote: I'm running ImageStream routers for the Internet distribution side of my network (2 edge routers, 2 core routers) and I'm extremely happy... This is a datacenter network and my customers are happy, I guess that's all that counts. In my opinion, I prefer to go with a open-source based solution because of pricing and customizability... I can build a script and load it into the equipment to give me any type of statistic I want... And I don't have to wait for a new IOS release. Note that imagestream is the worst of both worlds. it is ghetto like opensores but you don't get the source to fix it yourself if vendor is not being helpful. -alex
Zebra/linux device production networking?
Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_tc024.htm and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
(ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..) Unless you are absolutely certain of how routers need to work for your environment, and am willing to engineer your way out of problems, using this platform to achieve 99.x% uptime is quite not practical. Overall, this is a bad business decision, and if you quite had the clues to engineer most of the problems, you wouldn't be asking this question anyway ;) It's really a matter of lacking commercial support to route your traffic. If you can support yourself, then great, by all means go for it, and there are several operators running stable on cheap gears. If you can't support yourself, then you are opening up a can of worms. With that said, if you are looking to do one-router network for BGP, you may want to take a look at OpenBGPd, which is stable but currently lacks IGP support (though, openospfd is under works). Zebra is only stable when it's doing nothing or next to nothing. james
Re: Zebra/linux device production networking?
Linux routers are great for redundantly routing between your cable-modem and DSL at home. Using a linux router in production is a very very bad idea, although it may seem appealing to suits with no networking knowledge. I'm sure that other posters will provide you with many pages of reasons why linux routers suck, but I'll keep it short. 1. Mean Time Between Failures 2. OS exploits 3. Service/support Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment?
Re: Zebra/linux device production networking?
IMHO, it's a bad idea. A less intrusive alternative might be a FreeBSD based platform running Xorp/Quagga. Tiffany.On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: Greetings fellow nanogers,Long time lurker, first time poster (please, be gentle!).After looking at the archives, I didn't see this particular discussion,so here we go.First, a little background.. My CTO made my stomach curdle today when he announced that he wanted todo away with all our cisco [routers] and instead use Linux/zebra boxen.We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He hasinformed me that he has found many people who do this for their coredevices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device(core and/or regional, I'd be interested in both) in a production (read:99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did youfind out about after implementation?Has there been any discussion (or musings) of moving towards such asolution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in.Here's the article that started it all (this was featured on /., solikely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_tc024.htmand another:http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will ofcourse summarize to list or to individuals.(ps, particulars are deliberately not included.. I'm not looking foradvice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Albert Meyer wrote: 2. OS exploits One might argue that is an issue with any device. Cisco have their fair share of IOS updates due to security related bugs. Linux appears to have many, mostly due to the number of services that you can run. It's not like a Linux router is going to run Sendmail or Apache. David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgQHTIgPQWnLowkRAt4bAKDWOP4MOu3tnxTGxZDqPY+nlmS9DgCfZ1qi M8eUX6BsNNePrtEfT88Z/Aw= =pGLM -END PGP SIGNATURE-
RE: Zebra/linux device production networking?
Hi, I am also newbie poster so likewise plz be kind. I tend to agree with the comments made so far, however depending upon the business, budgets are not always available that might match the requirements and hence I can to some degree understand the use of such boxes for small organisations. I would be interested to know how many software (for want of a better description) routers are in live production in this kind of environment i.e. the 99.% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. Also does anyone have any peering policies which would exclude peers with software routers specifically, most have a requirement for the ability to support stable BGP peering but I have not seen any specific exclusions for such devices? Mark From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tiffany Snyder Sent: 06 June 2006 23:29 To: Nick Burke Cc: nanog@merit.edu Subject: Re: Zebra/linux device production networking? IMHO, it's a bad idea. A less intrusive alternative might be a FreeBSD based platform running Xorp/Quagga. Tiffany. On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_t c024.htm and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? Sure - I've done this before. We ran 7200s on the border (DS-3 interfaces for Linux didn't make sense at the time) and Linux boxes running all these features (plus some others) on the core. Worked flawlessly and the only downtime encountered over the two years it was running was during failover which took 5sec. Of course, the time invested in building it totally offset any savings, but that particular employer considered your time to be 'free', even though you could be billing instead, but that's a whole other argument. However, if I've got a Cisco router, in my city I can easily find 20 people in half an hour who I'd trust to get into my gear and work on it. I'd find another 50 if I went out 200miles. Linux on the other hand - Maybe three, including me. State wide, probably not even 20. I'm not talking RHCE people - I'm talking about people who can really troubleshoot kernel networking issues, device driver problems and so forth. Not easily accessible (or cheap) resources. Right now I've got a pair of Linux boxes (Debian based, 2.6 kernels) running Quagga (Zebra fork - I'd recommend it over Zebra) for BGP and OSPF, pulling two full loads. HSRP is provided with LinuxVirtualServer (aka heartbeat) and I'm doing dot1q with STP. No PVST support on Linux though. It all just works. Had a memory problem on one box, which killed it, but I've had that on plenty of Cisco gear too. None of the problems have really been 'Linux' related. 99% of them are user related, in that, I set an IP wrong, or I screw up a netmask - Usual kind of junk. Basically, if you're not comfortable with the idea of it, you're not comfortable supporting it. It'll cost leaps and bounds more supporting the environment compared to Cisco hardware. I have specific Linux expertise and experience which makes me go I can do that on Linux and have it work without problems, but also coming from a Cisco background I know where the line between being able to prove a point and making something that is manageable comes into play. Right now we're looking at building out a small POP in another building. I'm seriously considering a pair of Linux boxes running Quagga rather than 7200s that we'd normally go with. I can easily dump 3+ full loads on them, plus I can get gig connections on PCIe without having to fork out 10 grand on a NPE-G1. Am I going to do it? No idea. Technically, there is no issue. If I drop dead the day after it's built and someone new has to maintain it, then that's a potential problem. David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgdATIgPQWnLowkRAjPvAKDSoK/9kAZNjjQrix5aoMhM0v5fvACg7ilj 0fJYz8JLrH7iTjP49+XgmvE= =RAkO -END PGP SIGNATURE-
Re: Zebra/linux device production networking?
On Jun 6, 2006, at 4:42 PM, Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? We started out on a FreeBSD/Zebra routing solution for our company (content provider). While it did work acceptably for many years, it wasn't what I'd call robust. The router was a single P4 2.4GHz server. We had 4 GigE ports to 4 uplinks, each giving us a full BGP feed. Then two more GigE ports to our switches. We could route over 750mbps easily, without any packet loss or latency. The biggest issue we'd have was Zebra's single-threadedness. After a restart of bgpd, it would spend so much CPU time handling the BGP updates that it would get very very behind in processing BGP keepalives, and our sessions would time out before it had finished handling the initial burst. I'd have to shut down all sessions, then bring them up one at a time. It wasn't so much bgpd taking that much CPU, but bgpd not having very much left after the server was handling a few hundred mbps of traffic. Perhaps a dual CPU server would have worked better, but we never tried. There were also issues where you could get two zebra routers deadlocked - they'd both have many megabytes of BGP updates to send each other, and both would want to send a full update until completion before accepting any data in. Mucking with the kernel to allow TCP sockets to have a 16MB receive buffer helped, but still wasn't a cure. You're also giving up things like RIBs, fancy queuing/rate limiting, and any kind of hardware acceleration. Doing hundreds of megabits is easy, but software based routers seem to have trouble under DoS situations (lots of tiny packets) quicker. However, it was about as close to free as you could get. We re-used an old server, and only had to buy some 2 port ethernet cards. Support for Zebra is pretty iffy though. More often than not, I'd post a message to the Zebra mailing list to report a bug, and would get a Yeah, known bug! reply. The original author has all but abandoned development, leading to a fork called Quagga. Quagga is better (we still use it in a few places), but is still mostly a polished up Zebra. In the end, we needed to start pushing more traffic than we were able get our Zebra box to do. A couple 20+ minute outages during peak usage because of deadlocked bgpd processes helped my case that we needed to buy some Junipers instead. I know you're not giving specifics, but any kind of description of just how much traffic you're intending to push and how many ports you need would help in giving relevant advice. If you're talking about 1 BGP feed for 10mbps, I'd say go for it. If you're talking about a dozen sessions, and 2gbps of traffic... no way. Where you are between those is what really matters.
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark D. Kaye wrote: I would be interested to know how many software (for want of a better description) routers are in live production in this kind of environment i.e. the 99.% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. With the prevalence of Metro Ethernet, I'd think it's probably a pretty common thing. People run firewalls as routers (stuff like CheckPoint), which is basically Linux or FreeBSD, although not with EGP/IGP. Also does anyone have any peering policies which would exclude peers with software routers specifically, most have a requirement for the ability to support stable BGP peering but I have not seen any specific exclusions for such devices? MD5 authed BGP sessions might be an issue - At least with Linux it requires a kernel patch (works for me). I'd peered with plenty of big carriers with Linux stuff and they don't care. I probably have more issues with a carrier I peer with who uses Juniper and feeds me my prefixes at a rate of about 50/sec, rather than 2000/sec that I get from others using Cisco (My gear is Cisco in this instance) David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgiuTIgPQWnLowkRAo8eAJ9ZLANIku/rvRbRn5z5/kwbNnOspwCg5HfJ nUnzCg1xmcRc/4v3uiq1/eY= =bVnW -END PGP SIGNATURE-
Re: Zebra/linux device production networking?
On Tue, 6 Jun 2006, Nick Burke wrote: First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Having done exactly that previously, I wouldn't recommend it. While it will work, most of the time, reaching 99.999% will be a challenge. Amount of engineering time you will spend in order to reach that point (and to maintain your setup) will dwarf the cost of leasing proper equipment. Issues encountered: *) Performance under ddos: Linux routing stack is route-cache-based. That means, performance is a function of flows per second, and even small random src/dst ddos will kill you. Even when this is fixed, performance will be limited by pps - and the worst case performance of PC router is not as impressive as omg i can route 1gbit with p3/1ghz. In the end, worst case performance is what really matters, and it isn't all that awesome. *) Management: It takes certain amount of sysadmin time to manage each PC router (tools/etc). *) Integration: As it is not designed as a complete system, you will have little wierdnesses, such as, quagga not seeing kernel-installed routes, or netlink not being able to keep up with route updates, etc. All of those are fairly small things, but there are more than enough of them. *) Troubleshooting/continuity of operations: It takes two orders of magnitude more clue to troubleshoot zebra network - there are simply *lots* more things that can possibly go wrong - you don't worry just about your links breaking, you have to worry about your software being buggy. While any CCIE will most likely be able to troubleshoot and run a cisco-based network, pool of engineers sufficiently clued in a myriad of things that relate to troubleshooting of a PC router (ie. both network engineer, system admin, protocol engineer, kernel hacker, and at times, zebra-source-code-hacker) is far smaller. *) Maturity: While it has been improving, things like Quagga have still have stability issues and wierd issues that are resolved by killing ospfd. Because of a greater state of flux in such environment, you are likely to encounter things like oh, this bug is fixed in latest release - and then having to retest the new release which has completely different bugs. Yes, I know, you get that with proprietary vendors - but at least you get a benefit of *them* doing at least some amount of testing prior to release. *) Redundancy: Adding more redundancy to such a system is not likely to increase availability - in fact, it is likely to decrease availability because of added complexity and more things to break. Your problems are not likely to be the PC losing power (complete failure). Your problem will be things like zebra's idea of routing table being different from kernel's idea, zebra being unhappy after a transit flaps sucking up CPU time, leading to other things timing out, etc. Redundancy will excarcerbate these issues, making troubleshooting *harder*. So, in conclusion, if you have a large number of clued linux hackers who have nothing better to do, it may be a good idea. Otherwise, you'll realize you are spending far more on sysadmin time than you are saving on equipment cost. -- Alex Pilosov| DSL, Colocation, Hosting Services President | [EMAIL PROTECTED]877-PILOSOFT x601 Pilosoft, Inc. | http://www.pilosoft.com
Re: Zebra/linux device production networking?
(resent after getting on nanog-post) On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? I work for a company putting together an open source router platform. (Vyatta.com) We have a linux distro that is built off of XORP, but has plenty of enhancements that make it more friendly for a typical router jockey. It has dot1q support, bgp, ospf, rip, vrrp and many other goodies. We're currently going through UNH testing of protocol conformance. We are always looking for folks to test the software out and see how it suits their needs. (or not) Caveats: 1. Keep in mind that current sever hardware won't push line rate GigE at 64-bytes, but I find it quite reasonable as a candidate for the access layer. (t1/t3 and possibly oc3 termination) So don't expect it to perform to the same level as dedicated hardware solutions. A few hundred Mbps of inet traffic (not 64 byte frames) is reasonable. 2. Keep in mind that cheap PC hardware will result in bad MTBF. Your PC router hardware should be quality gear with redundancy if you can't tolerate any downtime. We believe there's a place for open source routing platforms, but it'll take some testing from the router community to solidify and verify the stacks. Want to help? --joel
