Re: [WISPA] Full BGP on RouterOS
Scott, I think others have already provided sufficient comments regarding the benefits hardware redundancy vs standby routers. We have both. Hardware redundancy in a GSR, router redundancy with another GSR in a separate geographical location. Continuous operation (hardware redundancy) in the event of a failure is our goal, but we're smart enough to realize we still need a backup for our backup (router redundancy). Look up Cisco's Nonstop Forwarding for more info regarding Travis' comment about the ability to lose an RSP/CPU and still forward traffic. -- Blake Covarrubias On Nov 3, 2010, at 17:26, Scott Reed scottr...@onlyinternet.net wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly redundant brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists jeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800
Re: [WISPA] Full BGP on RouterOS
You can plug one cable into 2 ports if it goes into a port that does bypassing. I am just guessing you can't have 1 Ethernet cable plugged into 2 ports on a single box either. When that port dies, all the redundancy in the box is worthless. Just for clarification, the 2 boxes ARE redundant. The amount of time required for fail-over does NOT define redundancy. Two different things. I agree, being fully redundant to reduce downtime to minutes and being fully redundant to reduce downtime to seconds or less are not the same from a business standpoint, but they are both redundant. On 11/3/2010 10:38 PM, Tom DeReggi wrote: You also have the problem where you cant have 1 ethernet cable plugged into two routers at the same time, unless you add switches in the front, which then adds complexity to setup and another point of failure. There is no question that there is value to a hardware redundant single server, in a mission ciritical environment. The question is, can one afford it, and is it cost justified. MANY CAN cost justify it. I cant. Its also possible for redundant hardware to fail, so there is also value to having a fail over second router. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: can...@believewireless.netp...@believewireless.net To: WISPA General Listwireless@wispa.org Sent: Wednesday, November 03, 2010 9:10 PM Subject: Re: [WISPA] Full BGP on RouterOS I think one of the main differences is BGP failover. With one box, your BGP session never drops. With two distinct servers, the session will drop and the second router will start it up. Then, when the primary comes back online, the session will drop again and restart. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- Scott Reed Sr. Systems Engineer GAB Midwest 1-800-363-1544 x2241 1-260-827-2241 Cell: 260-273-7239 WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Wait, you had ANY problem and MT told you to upgrade? No - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com On 11/2/2010 10:39 AM, Brad Belton wrote: Agreed. Breaking about once a month or possibly even more frequently than that is what we started to see with one of our v3.30 BGP routers. This was after flawless performance for many, many months. This is why we initially felt it might be a hardware issue. MikroTik Support response was simply to upgrade to current version (v4.11), so we did. That was about 50-60days ago I think? Since then the router hasn't had any trouble, so we're feeling pretty good about it. If it happens again or the hardware is/was the issue and it ultimately fails; we have an exact matching router racked up right beneath it ready to go. Eventually we may have Butch help us setup VRRP or some other method to tie the two routers together and keep both routers running all the time. We bring in multiple upstream GigE feeds from diverse geographical locations into our network, so losing a BGP router doesn't necessarily mean the network goes down...we just lose that particular peering point. Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:44 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing
Re: [WISPA] Full BGP on RouterOS
Hi Rubens, We've found Quagga to be rock solid with the typical application which is under a dozen peers. We did add a patch to prevent the never-emptying work queue backlog problem when multiple peers flap at the same time. I'm sure this is the problem the IXP folks ran into. Quagga is a very mature, stable RIP/OSPF/BGP platform which, given multi-threading capabilities, will scale to hundreds of peers. Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Rubens Kuhl Sent: Wednesday, November 03, 2010 3:51 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS On Wed, Nov 3, 2010 at 1:04 AM, Tom DeReggi wirelessn...@rapiddsl.net wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. Although it's a different scenario, the IXP folks beg to differ about Quagga reliability. When the number of peers is high, it flops miserably. Some of them moved to OpenBGPd, some of them to BIRD (http://bird.network.cz). None of them moved to XORP, Mikrotik's choice (and Vyatta's prior to switching to Quagga). If one have time, he or she should test all of the above... with limited time, I would favor testing BIRD first. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. And once you are comfortable with open-source border routing, you might want to take it to the next level by using hardware-based forwarding, with open-source software and gateware: http://www.netfpga.org/ Rubens WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1153 / Virus Database: 424/3234 - Release Date: 11/02/10 WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Thu, Nov 4, 2010 at 11:42 AM, Jeff Broadwick - Lists jeffl...@att.net wrote: Hi Rubens, We've found Quagga to be rock solid with the typical application which is under a dozen peers. We did add a patch to prevent the never-emptying work queue backlog problem when multiple peers flap at the same time. I'm sure this is the problem the IXP folks ran into. The Euro-IX folks developed some patches (or maybe some major code revision), you might want to take a look into that, but I guess you probably knew that already... Quagga is a very mature, stable RIP/OSPF/BGP platform which, given multi-threading capabilities, will scale to hundreds of peers. I saw some nasty bugs over the years with Quagga, and noticed the enormous effort required to maintain the old codebase; enough to make me always prefer something else. Every now and then a codebase seems to be more trouble maintaning than scrapping it altogether on the open-source world, and I'm pretty convinced that this time has come to Quagga. If you feel that codebase is worthwhile, I suggest investing a large amount of Imagestream revenues on restructuring it. I'll be glad to have Quagga as an option, again. Rubens WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Blake, I addressed the Quagga comments in a separate email. As to your comment about bridging, of course you don't need to bridge to use iBGP. I was specifically talking about using redundant routers in disparate locations. We have lots of customers who are using ImageStream routers for BGP redundancy.most using multiple routers. I'm glad you are happy with your Cisco routers. I didn't enter this thread to trash Cisco. If you have the right Cisco for the job, it will work just fine. Regards, Jeff ImageStream 800-813-5123 x106 _ From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 6:45 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I'm not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware.which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations.thus keeping your network running if something really nasty happens. I can't speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we've found it to be very stable, as our customers on-list have attested. It's one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists jeffl...@att.net wrote: I'm curious Travis.not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very
Re: [WISPA] Full BGP on RouterOS
For what it's worth, RouterOS 4.13 installed on the SuperMicro 5015A-H without a hitch (CF in a flash reader w/ netinstall). It actually booted on 2.9.50 (old flash), but lacked network interfaces. The performance difference between the P3 800 and the Atom 330 was minimal (as it relates to BGP lookups). Querying the routing tables was still in the 2-3 minute time range. Routing filter updates took about 30 seconds. Vyatta also installed just fine. However, even moderately complicated AS path regexs took only 2-3 seconds to complete, sometimes less. I couldn't find a query on the BGP table that took more than a couple of seconds. So it looks like the answer to my original question is...no, it's not normal and it's not the hardware, it's just RouterOS that doesn't perform very well in this area. On top of that, the ability to troubleshoot BGP routing issues on RouterOS appears to be significantly limited compared to Cisco and Cisco-like (e.g. Quagga) implementations. As far as the hardware goes, it appears to be performing quite well. Here are the parts if anyone is interested... 1x SuperMicro 5015A-H ($247) 2x DDR2 667 1G RAM ($20) 1x Super Talent 2.5 inch 16GB SATA SSD ($45) flamesuit I realize this isn't a Cisco GSR12000. We have and are using Cisco, Riverstone, etc. because they were best suited for a particular task. Each environment/problem warrants different possible solutions and variations based on personal preference, business needs, etc. /flamesuit Thanks again for all the information. This is obviously an complex problem with many possible solutions. -Kristian On Tue, 2010-11-02 at 17:37 -0700, Kristian Hoffmann wrote: On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
That is way cool, to have that much real redundancy in a router. How big is Big? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnson t...@ida.net To: WISPA General List wireless@wispa.org Sent: Tuesday, November 02, 2010 11:37 PM Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org
Re: [WISPA] Full BGP on RouterOS
Actually, answered own question... Saw picts on Google. Pretty sweet switch/router (12000 series), as long as its not sitting in an Equinix cage at $50/ 1U / month. Probably would costs $500-$700/mon to colo. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnson t...@ida.net To: WISPA General List wireless@wispa.org Sent: Tuesday, November 02, 2010 11:37 PM Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org
Re: [WISPA] Full BGP on RouterOS
Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Heard on Mikrotik forums 4.10 is more stable for BGP. When it crashes are there warning signs like high memory or CPU usage? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We had problems with 4.10 and went back to 4.3 and all is well. Haven't had a reason to upgrade to 4.11 but we'll probably just move straight to 5.0. On Wed, Nov 3, 2010 at 7:59 AM, Matt lm7...@gmail.com wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Heard on Mikrotik forums 4.10 is more stable for BGP. When it crashes are there warning signs like high memory or CPU usage? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We had problems with 4.10 and went back to 4.3 and all is well. Haven't had a reason to upgrade to 4.11 but we'll probably just move straight to 5.0. Heard from Mikrotik on forums that lookups with BGP could be related to OSPF and its fixed in v4.13 and latest v5RC. My understanding the OSPF bug caused lookups even if you were not using OSPF? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
It's about 30 tall and weighs about 100 pounds (literally). Travis Microserv On 11/3/2010 12:01 AM, Tom DeReggi wrote: That is way cool, to have that much real redundancy in a router. How big is Big? Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnsont...@ida.net To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 11:37 PM Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org
Re: [WISPA] Full BGP on RouterOS
I'm curious Travis.not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 _ From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org
Re: [WISPA] Full BGP on RouterOS
Hi, The higher end routers do their work in hardware with specific processors and memory for each function, which allows things like redundancy, speed, and hot-swap capabilities. I can pull any card from my Cisco router while it's running and put a different card in, configure it, and begin using it... all while the router is still routing. Realize I am talking about a 12000 series Cisco, that when brand new was probably over $250k fully loaded they are now on Ebay for $3k. Travis Microserv On 11/3/2010 9:04 AM, Jeff Broadwick - Lists wrote: I'm curious Travis...not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 *From:*wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] *On Behalf Of *Travis Johnson *Sent:* Tuesday, November 02, 2010 11:37 PM *To:* WISPA General List *Subject:* Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip
Re: [WISPA] Full BGP on RouterOS
Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists jeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance
Re: [WISPA] Full BGP on RouterOS
On Wed, Nov 3, 2010 at 1:04 AM, Tom DeReggi wirelessn...@rapiddsl.net wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. Although it's a different scenario, the IXP folks beg to differ about Quagga reliability. When the number of peers is high, it flops miserably. Some of them moved to OpenBGPd, some of them to BIRD (http://bird.network.cz). None of them moved to XORP, Mikrotik's choice (and Vyatta's prior to switching to Quagga). If one have time, he or she should test all of the above... with limited time, I would favor testing BIRD first. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. And once you are comfortable with open-source border routing, you might want to take it to the next level by using hardware-based forwarding, with open-source software and gateware: http://www.netfpga.org/ Rubens WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Hi Blake, I'm not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware.which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations.thus keeping your network running if something really nasty happens. I can't speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we've found it to be very stable, as our customers on-list have attested. It's one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 _ From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists mailto:jeffl...@att.net mailto:jeffl...@att.net mailto:jeffl...@att.net jeffl...@att.net wrote: I'm curious Travis.not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 _ From: mailto:wireless-boun...@wispa.org mailto:wireless-boun...@wispa.org mailto:wireless-boun...@wispa.org wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined
Re: [WISPA] Full BGP on RouterOS
When the number of peers is high, it flops miserably. I always wonder if that is an Education issue instead of a Quagga issue. Being connected to more peers enables more chances for bad routes sent or compatibility issues. One advantage of Quagga is its support base. Smarter people than I contribute to the code base. Because Quagga is used in many appliance's OS, and those manufacturer's programmers are likely to contribute to the Quagga source. Atleast Vyatta did. I'm not in an position technically to be able to comment on how Quagga compares to BIRD or OpenBGPd. But its good to know there are choices out there. (Note: I think Imagestream also offered GateD at one point as a choice, but I believe GateD no longer stacks up to Quagga) One negative thing about Quagga is that its authentication feature is not natively supported. So might need to run open, and use firewalling and filters to compensate, for lack of security. might want to take it to the next level by using hardware-based forwarding, with open-source software and gateware: http://www.netfpga.org/ Interesting to learn of and read. Although, I question at what point something like Hardware forwarding is really needed. With QuadCoreCPE or Dual Quad, PCI-E, NAPI, and I/0 scaling accross cores, just right there the forwarding speed is fantastic, into the multi-Gigbit (10gb). And as well, with newer XEON (5 series) AT/IO can add to it. (Although some work involved to enable such). Although it might not get full wire speed, it gets close. The advantage of sticking with Intel, is once again the support base. Writing drivers is often above the tech know how of the average ISP tech, and with Intel, a lot of the work is done for you by the commuity. But most importantly, that once Intel driver is selected, that you know there is a huge amount of hardware that will be available long term to use that code, without going back to the code writing drawing board. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Rubens Kuhl rube...@gmail.com To: WISPA General List wireless@wispa.org Sent: Wednesday, November 03, 2010 3:51 PM Subject: Re: [WISPA] Full BGP on RouterOS On Wed, Nov 3, 2010 at 1:04 AM, Tom DeReggi wirelessn...@rapiddsl.net wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. Although it's a different scenario, the IXP folks beg to differ about Quagga reliability. When the number of peers is high, it flops miserably. Some of them moved to OpenBGPd, some of them to BIRD (http://bird.network.cz). None of them moved to XORP, Mikrotik's choice (and Vyatta's prior to switching to Quagga). If one have time, he or she should test all of the above... with limited time, I would favor testing BIRD first. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. And once you are comfortable with open-source border routing, you might want to take it to the next level by using hardware-based forwarding, with open-source software and gateware: http://www.netfpga.org/ Rubens WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists jeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully
Re: [WISPA] Full BGP on RouterOS
OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly /redundant/ brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Listsjeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use
Re: [WISPA] Full BGP on RouterOS
You can *probably* do full tables on a pair of 1941's or 2900 Series Cisco's these days. With a pair of 1 U routers using VRRP or HSRP, you should be good to go. John On 11/2/2010 11:14 PM, Tom DeReggi wrote: Actually, answered own question... Saw picts on Google. Pretty sweet switch/router (12000 series), as long as its not sitting in an Equinix cage at $50/ 1U / month. Probably would costs $500-$700/mon to colo. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnsont...@ida.net To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 11:37 PM Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo
Re: [WISPA] Full BGP on RouterOS
I think one of the main differences is BGP failover. With one box, your BGP session never drops. With two distinct servers, the session will drop and the second router will start it up. Then, when the primary comes back online, the session will drop again and restart. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( Travis Microserv On 11/3/2010 6:26 PM, Scott Reed wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly /redundant/ brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From:wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Listsjeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From:wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works
Re: [WISPA] Full BGP on RouterOS
And, many of us in the middle of nowhere are still getting upstream links via telco circuits (such as OC3 and OC12). How do you terminate an OC12 into two separate boxes to run VRRP? You don't. Travis Microserv On 11/3/2010 6:26 PM, Scott Reed wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly /redundant/ brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From:wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Listsjeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS (Unix-based originally) to a hardened, purpose-built Linux distro (us, Mikrotik, Vyatta, whatever)? Regards, Jeff ImageStream 800-813-5123 x106 From:wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Travis Johnson Sent: Tuesday, November 02, 2010 11:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap
Re: [WISPA] Full BGP on RouterOS
You also have the problem where you cant have 1 ethernet cable plugged into two routers at the same time, unless you add switches in the front, which then adds complexity to setup and another point of failure. There is no question that there is value to a hardware redundant single server, in a mission ciritical environment. The question is, can one afford it, and is it cost justified. MANY CAN cost justify it. I cant. Its also possible for redundant hardware to fail, so there is also value to having a fail over second router. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: can...@believewireless.net p...@believewireless.net To: WISPA General List wireless@wispa.org Sent: Wednesday, November 03, 2010 9:10 PM Subject: Re: [WISPA] Full BGP on RouterOS I think one of the main differences is BGP failover. With one box, your BGP session never drops. With two distinct servers, the session will drop and the second router will start it up. Then, when the primary comes back online, the session will drop again and restart. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Wed, Nov 3, 2010 at 1:04 AM, Tom DeReggi wirelessn...@rapiddsl.net wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. Although it's a different scenario, the IXP folks beg to differ about Quagga reliability. When the number of peers is high, it flops miserably. Some of them moved to OpenBGPd, some of them to BIRD (http://bird.network.cz). None of them moved to XORP, Mikrotik's choice (and Vyatta's prior to switching to Quagga). Pretty sure Mikrotik is using none of those and instead rolled there own in there newer router OS releases. If one have time, he or she should test all of the above... with limited time, I would favor testing BIRD first. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Wed, 2010-11-03 at 19:53 -0600, Travis Johnson wrote: Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( What if the backplane is the problem? -- * Butch Evans * Professional Network Consultation* * http://www.butchevans.com/* Network Engineering * * http://store.wispgear.net/* Wired or Wireless Networks * * http://blog.butchevans.com/ * ImageStream, Mikrotik and MORE! * WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Wed, 2010-11-03 at 19:54 -0600, Travis Johnson wrote: And, many of us in the middle of nowhere are still getting upstream links via telco circuits (such as OC3 and OC12). How do you terminate an OC12 into two separate boxes to run VRRP? You don't. THIS is a more sensible argument. The other one (regarding what is/isn't redundant) is not. Both are redundant, but this method is redundant AND does the added feature of being connected to a (non-redundant) OC12 with a (non-redundant) OC12 interface. -- * Butch Evans * Professional Network Consultation* * http://www.butchevans.com/* Network Engineering * * http://store.wispgear.net/* Wired or Wireless Networks * * http://blog.butchevans.com/ * ImageStream, Mikrotik and MORE! * WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Powercode's MAXX does that...or so they say. I believe ImageStream says they can do this too. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Nov 3, 2010 at 9:53 PM, Travis Johnson t...@ida.net wrote: Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( Travis Microserv On 11/3/2010 6:26 PM, Scott Reed wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly *redundant* brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have attested. It’s one of our top applications. Regards, Jeff ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org wireless-boun...@wispa.org] On Behalf Of Blake Covarrubias Sent: Wednesday, November 03, 2010 11:31 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hardware redundancy, wire speed packet forwarding, support for more Interface types, and more widely tested stable software. I'll use a MikroTik, Linux, or BSD box as an aggregation router any day; terminate some VLANs, act as an MPLS CE, perform QoS marking, and participate in an OSPF area. Probably nothing more. The level of hardware redundancy wire-speed forwarding isn't there for my needs. If you're just knocking IOS, I realize it isn't the wave of the future. Cisco does too has developed IOS XR. Linux, MikroTik, and I'm sure Vyatta ImageStream are great platforms. They compete well with Cisco in some areas...others not so much. Use what's appropriate. -- Blake Covarrubias On Nov 3, 2010, at 8:04, Jeff Broadwick - Lists jeffl...@att.net jeffl...@att.net wrote: I’m curious Travis…not looking for an argument. What specifically do you think is superior in IOS
Re: [WISPA] Full BGP on RouterOS
Although it's a different scenario, the IXP folks beg to differ about Quagga reliability. When the number of peers is high, it flops miserably. Some of them moved to OpenBGPd, some of them to BIRD (http://bird.network.cz). None of them moved to XORP, Mikrotik's choice (and Vyatta's prior to switching to Quagga). Pretty sure Mikrotik is using none of those and instead rolled there own in there newer router OS releases. - Mikrotik ROS 2.9x was Quagga-based (you could do a telnet 127.0.0.1 2601, for instance) ; - Mikrotik ROS 3.15 was XORP-based according to Mikrotik; they told that to a customer that was facing issues with that version ; - On the Mikrotik wiki you can find info that PIM-SM Multicast code is from XORP, although they don't say on the wiki that BGP or OSPF were XORP's ; - The minutes long CPU-hog was a bug described at Vyatta forums while they were using XORP, and suddenly the same bug appears on a multitude of Mikrotik ROS versions. The proof is left as an exercise for the reader. Has been XORP part of ROS at same point ? Pretty sure it was. Is still XORP-based on 4.x, 5.0rc ? I don't know. Educated guess: XORP up to 4.something, something else on 4.later or 5.0rc. Rubens WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Hot Swap is hard to fully accomplish with PCs. Everyone needs a plan for how maintenance will occur with minimal downtime. For example, its prettty easy to buy a nice Rack case with redundant PS, but how do you replace an overheating CPU? A standard Rack PC does not have HotSwap CPUs, and it is inevitable that sooner or later the Heatsink fan will fail or heat sink grease will harden. And how does one troubleshoot that, on a live router? That is the negative of a Linux self made Rack PC. But again, thats the reason for a hot spare router to put in place, and a reason for scheduled maintenance to occur every couple years after hours, when a 60 second outage is acceptable. As ISPs start to become gloabal ISPs opperating in multiple time zones, it becomes tougher, to find good times to do maintenance, but I dont think most WISPs are at that stage where it matters that much. When it does matter that much, I'd argue the WISP should have both hardware redundancy and router redundancy. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Josh Luthman To: WISPA General List Sent: Wednesday, November 03, 2010 11:26 PM Subject: Re: [WISPA] Full BGP on RouterOS Powercode's MAXX does that...or so they say. I believe ImageStream says they can do this too. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Nov 3, 2010 at 9:53 PM, Travis Johnson t...@ida.net wrote: Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( Travis Microserv On 11/3/2010 6:26 PM, Scott Reed wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly redundant brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience. If its not reliable we replace it. We don't have a problem paying for reliability. -- Blake Covarrubias On Nov 3, 2010, at 1:16 PM, Jeff Broadwick - Lists wrote: Hi Blake, I’m not sure what sort of speeds you think Linux limits out at, but I believe you might be surprised at how much throughput you can get. We generally blow the doors off of the VXRs and down. There are two different ways of getting hardware redundancy. One is with a massively expensive single box, like the Cisco. The other is to set up redundant hardware…which is particularly good in a BGP application. You can have a relatively inexpensive router on each circuit, set up iBGP and VRRP between the boxes, and BGP between the peers. That way, if you lose anything, all the in and outbound traffic fails to the other unit(s). This also allows for geographic separation of the routers. If you can bridge between the routers, you can have them in completely different locations…thus keeping your network running if something really nasty happens. I can’t speak for the other companies, but ImageStream has been handling BGP for around 10 years. We use Quagga currently and we’ve found it to be very stable, as our customers on-list have
Re: [WISPA] Full BGP on RouterOS
Then you move the cards into the spare chassis you have sitting 3ft away in another rack and boot up and go... :) However, I have NEVER heard of a Cisco 12000 series backplane failing. EVER. Can't say that for an X86 based anything... they fail all the time... cards, system boards, processors, memory, power supplies, etc. Don't get me wrong here... we love Mikrotik running on our X86 systems we currently have 20 or 30 of them running the heart of our network... including one directly behind our Cisco 12008 that does all of our routing and deactivation stuff... and handles 450Mbps x 200Mbps on a daily basis... :) The point was, running RouterOS on a device taking multiple BGP feeds... which I would never do... Cisco still owns the BGP space... and my next choice would be Imagestream. Travis Microserv On 11/3/2010 8:45 PM, Butch Evans wrote: On Wed, 2010-11-03 at 19:53 -0600, Travis Johnson wrote: Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( What if the backplane is the problem? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Agreed. Whenever we buy any major component for our network (big routers, licensed links, big switches, etc.) we always buy a spare to go with it (if we don't already have one). Example... our main backbone switch is a Cisco 3550-12T. All of our traffic (currently 450Mbps x 200Mbps) goes through this switch so we can mirror some of the ports for filtering and traffic monitoring. When we first purchased the switch, we bought 2 of them. We mounted the 2nd one directly above the first in our rack, and configured it exactly the same. So, if that switch ever dies, we simply move a few cables and we are back up and going... and at the same time, if we ever need to upgrade firmware, we can do it on the 2nd one, test it, and then move it into production and upgrade the 1st one with about 5 seconds of total downtime while we move cables. Always, always have spares of your critical equipment. :) Travis Microserv On 11/3/2010 10:54 PM, Tom DeReggi wrote: Hot Swap is hard to fully accomplish with PCs. Everyone needs a plan for how maintenance will occur with minimal downtime. For example, its prettty easy to buy a nice Rack case with redundant PS, but how do you replace an overheating CPU? A standard Rack PC does not have HotSwap CPUs, and it is inevitable that sooner or later the Heatsink fan will fail or heat sink grease will harden. And how does one troubleshoot that, on a live router? That is the negative of a Linux self made Rack PC. But again, thats the reason for a hot spare router to put in place, and a reason for scheduled maintenance to occur every couple years after hours, when a 60 second outage is acceptable. As ISPs start to become gloabal ISPs opperating in multiple time zones, it becomes tougher, to find good times to do maintenance, but I dont think most WISPs are at that stage where it matters that much. When it does matter that much, I'd argue the WISP should have both hardware redundancy and router redundancy. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - *From:* Josh Luthman mailto:j...@imaginenetworksllc.com *To:* WISPA General List mailto:wireless@wispa.org *Sent:* Wednesday, November 03, 2010 11:26 PM *Subject:* Re: [WISPA] Full BGP on RouterOS Powercode's MAXX does that...or so they say. I believe ImageStream says they can do this too. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Nov 3, 2010 at 9:53 PM, Travis Johnson t...@ida.net mailto:t...@ida.net wrote: Having two routers talking to each other is not the same as a single router with redundant parts. I can pull the CPU card from my Cisco and the box never misses a single packet because the 2nd CPU card is in the same box. Same with the route processor cards. Same with the power supplies. If you have two boxes doing VRRP, and BGP, if the power supply goes out of a box, how long before the 2nd box could fully take over? 30 seconds? 60 seconds? :( Travis Microserv On 11/3/2010 6:26 PM, Scott Reed wrote: OK, elaborate on how 2 distinct identical boxes is not hardware redundancy. I think by the definition of redundancy, it is 100%. Webster: characterized by similarity or repetition a group of particularly /redundant/ brick buildings On 11/3/2010 6:45 PM, Blake Covarrubias wrote: Jeff, VXRs and down. Not GSR's and up. I wasn't entirely clear in my last message. Like Travis I was also commenting about the Cisco GSR / 12000 platform. I'm well aware of the performance of a Linux box compared to a VXR. We run a few VXR routers in our network in addition to GSR's, BSD routers, and MikroTik. What you're describing really isn't true hardware redundancy. I'm also well aware of BGP and its use in a multi-homed environment. We have two separate GSRs acting as our edge routers. One in California, one in Arizona. Both routers have multiple eBGP peers, and run iBGP between them. They're connected by a series of licensed microwave radios with about 155mbps of bandwidth between the two. We'll be supplementing that link with a dedicated GigE fiber link in the coming months. I'm not sure what you're getting at regarding bridging between two connections. There's no requirement to run a bridged network in order to operate iBGP. I have no doubt Quagga works well in some BGP applications. We don't use it because we have requirements for performance uptime which a Linux/BSD box cannot currently meet. We provide voice (TDM) and data services for companies in various industries such as mining, manufacturing, aerospace, defense, energy, cellular, and even other ISPs. We literally cannot afford to wrestle with the issues others on this list experience
Re: [WISPA] Full BGP on RouterOS
On Wed, 2010-11-03 at 22:59 -0600, Travis Johnson wrote: Then you move the cards into the spare chassis you have sitting 3ft away in another rack and boot up and go... :) My only point was that all that redundancy, which I think is a GOOD thing, is only redundant to a point. At some point in the system, there is a point of failure. However, I have NEVER heard of a Cisco 12000 series backplane failing. EVER. Nor have I. That wasn't really the point I was making anyway. The point, as stated above, is that there IS a point of failure in the system, even if it is a rare failure. Don't get me wrong, the router you have sounds like a great bargain. I am not knocking that at all. Can't say that for an X86 based anything... they fail all the time... cards, system boards, processors, memory, power supplies, etc. I'd say that this is a bit of an exaggeration. The point was, running RouterOS on a device taking multiple BGP feeds... which I would never do... Cisco still owns the BGP space... and my next choice would be Imagestream. In general, I don't recommend ROS for BGP when you need anything beyond basic functionality. The only difference between your choice and mine is which one we put as first choice, which, for me, would be ImageStream. Except that Cisco wouldn't likely be my second choice, either...I'd be more likely to go with Juniper and then Cisco. -- * Butch Evans * Professional Network Consultation* * http://www.butchevans.com/* Network Engineering * * http://store.wispgear.net/* Wired or Wireless Networks * * http://blog.butchevans.com/ * ImageStream, Mikrotik and MORE! * WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 From: Kristian Hoffmann kh...@fire2wire.com Sent: Friday, October 29, 2010 11:50 AM To: WISPA General List wireless@wispa.org Subject: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). I don't suppose someone could share the Mikrotik BGP config? We are looking at moving to BGP on the Mikrotik with a couple providers. May need to weight one heavier then the other eventually somehow. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 *From*: Kristian Hoffmann kh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General List wireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmann kh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General List wireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE -- -- WISPA Wants You! Join today! http://signup.wispa.org/ -- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- -- WISPA Wants You! Join today! http://signup.wispa.org/ -- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org
Re: [WISPA] Full BGP on RouterOS
Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Belton b...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmann kh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General List wireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE -- -- WISPA Wants You! Join today! http://signup.wispa.org/ -- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- -- WISPA Wants You! Join today! http://signup.wispa.org/ -- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org
Re: [WISPA] Full BGP on RouterOS
Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. When it crashes does it reboot itself? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE -- -- WISPA Wants You! Join today! http://signup.wispa.org/ -- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- -- WISPA Wants You! Join today! http
Re: [WISPA] Full BGP on RouterOS
Look into Imagestream for BGP. Very stable and less expensive than Cisco. IMHO you want something running the edge of your network to have support and software updates. If you go with Cisco plan on the extra expense of Smartnet. Having said that I have ran BGP on Imagestream in the 3.X range. I believe the highest any BGP routers I have are 3.30. The reason being is you don¹t have to run the latest greatest for BGP. It¹s a very simple protocol actually. These are x86 P4 machines receiving partial routes, not full. Justin -- Justin Wilson j...@mtin.net http://www.mtin.net/blog xISP News http://www.twitter.com/j2sw Follow me on Twitter Wisp Consulting Tower Climbing Network Support From: Faisal Imtiaz fai...@snappydsl.net Reply-To: fai...@snappydsl.net, WISPA General List wireless@wispa.org Date: Tue, 02 Nov 2010 10:43:43 -0400 To: WISPA General List wireless@wispa.org Subject: Re: [WISPA] Full BGP on RouterOS This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed
Re: [WISPA] Full BGP on RouterOS
So far the box running 5.0RC1 has been rock solid. Its a routermaxx 1200. So it has to run 5.x, No memory leak issues. Both of our BGP routers have 0 problems. No strange reboots, No strange anything. One running 4.4 one with 5.0rc1. We had problems in 4.1 I belive with it idling out a session or two every few hours. Since we put 4.4+ on our routers, We have been more stable then the our upstreams. Nick Olsen Network Operations (855) FLSPEED x106 From: Faisal Imtiaz fai...@snappydsl.net Sent: Tuesday, November 02, 2010 10:43 AM To: WISPA General List wireless@wispa.org Subject: Re: [WISPA] Full BGP on RouterOS This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian
Re: [WISPA] Full BGP on RouterOS
Anyone doing Vyatta for Edge? Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. 787.273.4143 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Justin Wilson Sent: Tuesday, November 02, 2010 10:52 AM To: fai...@snappydsl.net; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Look into Imagestream for BGP. Very stable and less expensive than Cisco. IMHO you want something running the edge of your network to have support and software updates. If you go with Cisco plan on the extra expense of Smartnet. Having said that I have ran BGP on Imagestream in the 3.X range. I believe the highest any BGP routers I have are 3.30. The reason being is you don't have to run the latest greatest for BGP. It's a very simple protocol actually. These are x86 P4 machines receiving partial routes, not full. Justin -- Justin Wilson j...@mtin.net http://www.mtin.net/blog - xISP News http://www.twitter.com/j2sw - Follow me on Twitter Wisp Consulting - Tower Climbing - Network Support From: Faisal Imtiaz fai...@snappydsl.net Reply-To: fai...@snappydsl.net, WISPA General List wireless@wispa.org Date: Tue, 02 Nov 2010 10:43:43 -0400 To: WISPA General List wireless@wispa.org Subject: Re: [WISPA] Full BGP on RouterOS This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed
Re: [WISPA] Full BGP on RouterOS
We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations
Re: [WISPA] Full BGP on RouterOS
Agreed. Breaking about once a month or possibly even more frequently than that is what we started to see with one of our v3.30 BGP routers. This was after flawless performance for many, many months. This is why we initially felt it might be a hardware issue. MikroTik Support response was simply to upgrade to current version (v4.11), so we did. That was about 50-60days ago I think? Since then the router hasn't had any trouble, so we're feeling pretty good about it. If it happens again or the hardware is/was the issue and it ultimately fails; we have an exact matching router racked up right beneath it ready to go. Eventually we may have Butch help us setup VRRP or some other method to tie the two routers together and keep both routers running all the time. We bring in multiple upstream GigE feeds from diverse geographical locations into our network, so losing a BGP router doesn't necessarily mean the network goes down...we just lose that particular peering point. Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:44 AM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 9:21 AM, Nick Olsen wrote: We have two full tables running on mikrotik, in two different locations. Running that command /ip route print count-only where bgp-as-path=1234 Replacing the AS with 33363 (local cable company). Doesn't work on either of our routers for some reason (MT 5.0rc1 or 4.4). Our router running a core 2 2.93ghz can take two full feeds gets all the routes in about 4 seconds, And cpu load is idle about 13 seconds later. However making changes with routing filters take anywhere from 10seconds to 2 minutes depending on what its doing. Nick Olsen Network Operations (855) FLSPEED x106 -- -- *From*: Kristian Hoffmannkh...@fire2wire.com *Sent*: Friday, October 29, 2010 11:50 AM *To*: WISPA General Listwireless@wispa.org *Subject*: [WISPA] Full BGP on RouterOS Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all
Re: [WISPA] Full BGP on RouterOS
I haven't tried using MT for my eBGP feeds, but with so many people having issues I'm not even interested in giving it a shot. Buy a Cisco or Juniper. I've run BGP on refurbished Cisco 7500's and GSR's for the past few years with zero issues. My routers have at least two eBGP peers, full tables, and zero problems. Route filters take seconds to apply, as does searching the BGP routing table. For those having issues, why deal with the headaches of MT? Go with something that's solid. MikroTik is great is some areas; BGP doesn't sound like one of them. Use the best tool for the job. -- Blake Covarrubias On Nov 2, 2010, at 8:35, jp j...@saucer.midcoast.com wrote: We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has the Mikrotik been running full BGP with the two providers ? (I read about a memory leak issues, is that why you are using 5.0rc1 ?) We have been considering getting a Mikrotik for such use. Thanks. Faisal Imtiaz
Re: [WISPA] Full BGP on RouterOS
Hi J.P. and others, I have a dog in this fight (I work for ImageStream), but this comment applies to just about any situation/hardware. Set up redundant routers using BGP and VRRP (HSRP in Cisco.although apparently they support VRRP also). If you can't afford to be down, you need to find a way to get another router. If your upstreams are Ethernet, this is pretty simple, and not terribly expensive (depending on the hardware you choose). You can peer with different types/speeds of circuits. You'll need to do some preliminary work to manage the flows so that you don't kill the smaller feeds, but it's very doable. Regards, Jeff Broadwick ImageStream 800-813-5123 x106 _ From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of jp Sent: Tuesday, November 02, 2010 11:35 AM To: fai...@snappydsl.net; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens of thousands of dollars ahead. That coupled with building a network that isn't solely dependent on any single point of failure further reduces the crisis when a core router fails. Things break...doesn't matter if MikroTik, ImageStream, Cisco or Juniper makes it. ALL things break eventually, so plan for it! Best, Brad -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 9:11 AM To: n...@brevardwireless.com; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Nick, How stable has
Re: [WISPA] Full BGP on RouterOS
Hi Jeff, Glad to see you jump into this.. Would you feel comfortable is sharing what Model Image Stream will do the following:- be able to handle full bgp feeds from multiple providers ( 2gig Ram ?) and be able to handle 300 - to a GIG of traffic ? Rough cost of unit will be great to know as well. (Gig E feeds, copper Fiber). And How Stable is the Imagestream in running this type of a configuration. Regards. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 12:04 PM, Jeff Broadwick - Lists wrote: Hi J.P. and others, I have a dog in this fight (I work for ImageStream), but this comment applies to just about any situation/hardware. Set up redundant routers using BGP and VRRP (HSRP in Cisco…although apparently they support VRRP also). If you can’t afford to be down, you need to find a way to get another router. If your upstreams are Ethernet, this is pretty simple, and not terribly expensive (depending on the hardware you choose). You can peer with different types/speeds of circuits. You’ll need to do some preliminary work to manage the flows so that you don’t kill the smaller feeds, but it’s very doable. Regards, Jeff Broadwick ImageStream 800-813-5123 x106 *From:*wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] *On Behalf Of *jp *Sent:* Tuesday, November 02, 2010 11:35 AM *To:* fai...@snappydsl.net; WISPA General List *Subject:* Re: [WISPA] Full BGP on RouterOS We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related, however moving it to v4.11 seems to have resolved the problem. (knock on wood) Bottom line is given the price of a beefy MikroTik router vs. buying an Imagestream or Cisco that is equivalent we can have hot standby spares on hand and still be thousands if not tens
Re: [WISPA] Full BGP on RouterOS
For those interested in BGP and VRRP, take look at this thread from the Vyatta forums. http://www.vyatta.org/forum/viewtopic.php?p=4213sid=0b9f48079b1388c4fb722704ac6221ae Its not hitless, stateful failover, but this method will work and provides probably the best failover you're going to see given the constraints of the BGP protocol. You're still going to be subject to the same latency in your failover scheme as Vyatta; VRRP, BGP, firewall sessions. As long as your provider will run dual BGP feeds to you from the same router on their side you can create this setup. The config is relatively straight forward. If anyone is interested in seeing this configured under MikroTik hit me up off list and I'll come up with a sample config. -- Blake Covarrubias On Nov 2, 2010, at 9:04 AM, Jeff Broadwick - Lists wrote: Hi J.P. and others, I have a dog in this fight (I work for ImageStream), but this comment applies to just about any situation/hardware. Set up redundant routers using BGP and VRRP (HSRP in Cisco…although apparently they support VRRP also). If you can’t afford to be down, you need to find a way to get another router. If your upstreams are Ethernet, this is pretty simple, and not terribly expensive (depending on the hardware you choose). You can peer with different types/speeds of circuits. You’ll need to do some preliminary work to manage the flows so that you don’t kill the smaller feeds, but it’s very doable. Regards, Jeff Broadwick ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of jp Sent: Tuesday, November 02, 2010 11:35 AM To: fai...@snappydsl.net; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue, Nov 2, 2010 at 10:23 AM, Brad Beltonb...@belwave.com wrote: We've been running BGP with MikroTik for quite some time now. It hasn't been flawless by any stretch, but ever since late v2.8 or early v2.9 we haven't had much trouble with it. We running v3.30 on two routers with two full feeds each and a third running v4.11 with two full feeds. All of these routers have a handful of downstream BGP peers that we are also delivering full tables to. So far I think v4.11 might be the best, but we don't have as much time on that version as we do with v3.30. The only reason we moved one of our routers from v3.30 to v4.11 was because we had an unusual hang with that particular router. We weren't sure if it was hardware or OS related
Re: [WISPA] Full BGP on RouterOS
Hi Faisal, I'll be happy to get you specific pricing off-list. I'm not totally sure how far I am allowed to go on-list. The specific router I would recommend would be dependent on a couple factors: Average packet size - If this is a VoIP heavy application, the load on the router will be much higher. Type of circuit - If the GigE is delivered via copper, I have more options. If it is delivered via fiber that puts me in a higher end box) If the packets are of normal ISP size, and the circuits are delivered via copper GigE, our Rebel router would be fine for the application. 2 of the routers, with a Gig of RAM each (plenty for 2 feeds), and with a year of support/warranty would be around $5K. There are additional support and hardware options that are available also. Regards, Jeff 800-813-5123 x106 _ From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Faisal Imtiaz Sent: Tuesday, November 02, 2010 12:11 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS Hi Jeff, Glad to see you jump into this.. Would you feel comfortable is sharing what Model Image Stream will do the following:- be able to handle full bgp feeds from multiple providers ( 2gig Ram ?) and be able to handle 300 - to a GIG of traffic ? Rough cost of unit will be great to know as well. (Gig E feeds, copper Fiber). And How Stable is the Imagestream in running this type of a configuration. Regards. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 12:04 PM, Jeff Broadwick - Lists wrote: Hi J.P. and others, I have a dog in this fight (I work for ImageStream), but this comment applies to just about any situation/hardware. Set up redundant routers using BGP and VRRP (HSRP in Cisco.although apparently they support VRRP also). If you can't afford to be down, you need to find a way to get another router. If your upstreams are Ethernet, this is pretty simple, and not terribly expensive (depending on the hardware you choose). You can peer with different types/speeds of circuits. You'll need to do some preliminary work to manage the flows so that you don't kill the smaller feeds, but it's very doable. Regards, Jeff Broadwick ImageStream 800-813-5123 x106 *From:*wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] *On Behalf Of *jp *Sent:* Tuesday, November 02, 2010 11:35 AM *To:* fai...@snappydsl.net; WISPA General List *Subject:* Re: [WISPA] Full BGP on RouterOS We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks get bigger and thus more complex, it's essential to avoid unreliable situations, no matter what the cost savings. Say you have 60 devices and each one has a once-a-month breakage. That's 2 outages a day, enough to drive staff and customers crazy. Reduce that to twice a year outage per device and that staffer can get something done again. On Tue, Nov 02, 2010 at 10:43:43AM -0400, Faisal Imtiaz wrote: This is exactly what I am concerned with. Things breaking once in a while is not an issue.. Things breaking once every month or few weeks is not going to be acceptable from our users.. Trying to determine if this is a 'feature' or a short term 'bug'. Cisco's and Junipers, get a premium even in the used market place, but the primary reason for it is stability... Any other that can chime in with their experiences ? Many thanks in advance. Faisal Imtiaz Snappy Internet Telecom On 11/2/2010 10:32 AM, Chuck Hogg wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. Regards, Chuck On Tue
Re: [WISPA] Full BGP on RouterOS
No, we have to hard reboot it. Regards, Chuck On Tue, Nov 2, 2010 at 10:39 AM, Matt lm7...@gmail.com wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. When it crashes does it reboot itself? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
No, we have to hard reboot it. Regards, What kind of hardware is it running on? Have you checked memory? On Tue, Nov 2, 2010 at 10:39 AM, Matt lm7...@gmail.com wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. When it crashes does it reboot itself? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Although a P3 800 is not something we could call powerful these days, what you've seen is connected to software, not hardware. Since Mikrotik replaced Quagga with XORP in ROS 3.x, a good number of users report minutes of high CPU in a full-routing environment. Does not happen for everyone, but happens to most of them. Rubens On Fri, Oct 29, 2010 at 1:51 PM, Kristian Hoffmann kh...@fire2wire.com wrote: Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
I'll fill in some more details for Chuck.. We started out on a RB1000 with a single bgp peer. All worked fine for a long time. We then added a second bgp peer and switched to full tables tables and it started leaking memory and would lock after a few weeks to a couple months of running. After the third bgp peer with full tables it would lockup after several days to a couple weeks. We upgraded the memory on the RB1000 thinking that more might help.. It didn't.. We tried several RB1000's and eventually upgraded to an Axiomtek NA820 x86 system a few months ago. Sunday, after 62 days uptime the x86 system locked for the first time. We've tried ROS 3.1x to 4.11 all with the same results. I've sent a number of supouts to Mikrotik at various days of uptime before it locked. I've hooked a console cable to the RB1000 after it locked and was not able to get any response from it. I need to leave a monitor hooked to the x86 system to see if it is kernel panicing or just freezing. I have another x86 system that I think I'm going to load freebsd on it and just run Quagga. I don't want to spend the money on a cisco to handle 3+ full bgp feeds and 300+mbit of traffic. I've considered splitting up the bgp peers with each on their own router and running ibgp between them, but I'm still facing the same problem of having 2 full bgp tables on each router. The only way I've been able to minimize the time between lockups is to filter everything but default routes from my upstreams. Gerard Dupont Shelby Broadband On Tue, Nov 2, 2010 at 3:50 PM, Matt lm7...@gmail.com wrote: No, we have to hard reboot it. Regards, What kind of hardware is it running on? Have you checked memory? On Tue, Nov 2, 2010 at 10:39 AM, Matt lm7...@gmail.com wrote: Our MikroTik BGP router keeps crashing about once every month or so...sometimes sooner, sometimes later. We are using full BGP tables and 4.11 currently. When it crashes does it reboot itself? WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Tue, Nov 02, 2010 at 05:23:40PM -0400, Gerard Dupont wrote: I have another x86 system that I think I'm going to load freebsd on it and just run Quagga. I don't want to spend the money on a cisco to handle 3+ full bgp feeds and 300+mbit of traffic. bsdrp.org? or a full FreeBSD install? I just configured my first bsdrp install. It's not in production yet, but configuring it didn't suck. I'm a cisco guy so I feel more at home in Quagga. I setup all but the primary LAN IP in quagga rather than rc.conf so that I can be closer to seeing/changing everything from one place. We also have a few older ImageStream Transport routers. The are pretty nice to work with. They won't handle 300Mbps of traffic but the work great at our tower sites. We only have about 150Mbps traffic total anyway. Just realized they even have mtr installed. The management interface is probably better than the BSDRP methods for non-FreeBSD gearheads. I still need to try a Vyatta system. -- Scott LambertKC5MLE Unix SysAdmin lamb...@lambertfam.org WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Image stream, excellent price and support -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Kristian Hoffmann Sent: Tuesday, November 02, 2010 5:37 PM To: WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On 11/02/2010 08:37 PM, Kristian Hoffmann wrote: On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian I don't know I find quagga runnning on a Ubuntu server to be a piece of cake and having all the networking tools of Linux makes it nice when I need to troubleshot a network problem. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Nobody asked, but here it is. A sample BGP and VRRP config. I tested this with a few MetaROUTER VM's and it all worked. Hopefully I don't have any typos. I implemented the config to mirror that shown in the picture from the thread on the Vyatta forum. If you need assistance adapting this to your exact config let me know and I'll be happy to help. ### Router 1 ### /interface vrrp add interface=ether1 name=vrrp1 vrid=1 priority=254 authentication=ah password=somepass /ip address add interface=ether1 address=192.168.111.2/29 add interface=vrrp1 address=192.168.111.4/29 /routing bgp instance set default as=65001 router-id=192.168.111.2 /routing filter add chain=set-next-hop locally-originated-bgp=yes set-out-nexthop=192.168.111.4 add chain=set-next-hop locally-originated-bgp=yes action=accept add chain=set-next-hop action=reject comment=Deny any other routes /routing bgp peer add instance=default name=upstream remote-address=192.168.111.1 remote-as=65000 update-source=192.168.111.2 nexthop-choice=propagate out-filter=set-next-hop /routing bgp network add network=192.168.110.0/24 synchronize=no ### Router 2 ### /interface vrrp add interface=ether1 name=vrrp1 vrid=1 priority=100 authentication=ah password=somepass /ip address add interface=ether1 address=192.168.111.3/29 add interface=vrrp1 address=192.168.111.4/29 /routing bgp instance set default as=65001 router-id=192.168.111.3 /routing filter add chain=set-next-hop locally-originated-bgp=yes set-out-nexthop=192.168.111.4 add chain=set-next-hop locally-originated-bgp=yes action=accept add chain=set-next-hop action=reject comment=Deny any other routes /routing bgp peer add instance=default name=upstream remote-address=192.168.111.1 remote-as=65000 update-source=192.168.111.3 nexthop-choice=propagate out-filter=set-next-hop /routing bgp network add network=192.168.110.0/24 synchronize=no -- Blake Covarrubias On Nov 2, 2010, at 9:13 AM, Blake Covarrubias wrote: For those interested in BGP and VRRP, take look at this thread from the Vyatta forums. http://www.vyatta.org/forum/viewtopic.php?p=4213sid=0b9f48079b1388c4fb722704ac6221ae Its not hitless, stateful failover, but this method will work and provides probably the best failover you're going to see given the constraints of the BGP protocol. You're still going to be subject to the same latency in your failover scheme as Vyatta; VRRP, BGP, firewall sessions. As long as your provider will run dual BGP feeds to you from the same router on their side you can create this setup. The config is relatively straight forward. If anyone is interested in seeing this configured under MikroTik hit me up off list and I'll come up with a sample config. -- Blake Covarrubias On Nov 2, 2010, at 9:04 AM, Jeff Broadwick - Lists wrote: Hi J.P. and others, I have a dog in this fight (I work for ImageStream), but this comment applies to just about any situation/hardware. Set up redundant routers using BGP and VRRP (HSRP in Cisco…although apparently they support VRRP also). If you can’t afford to be down, you need to find a way to get another router. If your upstreams are Ethernet, this is pretty simple, and not terribly expensive (depending on the hardware you choose). You can peer with different types/speeds of circuits. You’ll need to do some preliminary work to manage the flows so that you don’t kill the smaller feeds, but it’s very doable. Regards, Jeff Broadwick ImageStream 800-813-5123 x106 From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of jp Sent: Tuesday, November 02, 2010 11:35 AM To: fai...@snappydsl.net; WISPA General List Subject: Re: [WISPA] Full BGP on RouterOS We use MT BGP internally on our network; not full feeds. 1400 routes on one server. Works great for that; no reliability issues in every day operation. No problems with 12 month uptimes. I have seen some minor issues in 3.30 where if you remove a peer prior to disabling first, it can jam things up in terms of stale routes, but done in the right order it's fine. Once a month crash is not acceptable reliability for our uplink. Once in 6 months is OK if it's self-correcting. I've used Cisco for about ten years for my uplink with BGP, and it has been good as long as we didn't run out of memory or have a rare hardware problem. We kept a complete working spare system in place for parts and didn't have a Smartnet contract. I had a smartnet once for a smaller router, but it was so much trouble getting it and using it for just one router, I never renewed it. We have a Juniper j2350 with aftermarket ram now replacing a Cisco 7507. It's been as/more reliable than the Cisco and is 2u instead of dorm-fridge size, gigabit ports instead of 100mbps ports. Tech support was excellent. I'll be buying a spare or bigger one for backup. I have not tried Imagestream, but don't doubt the wide variety of other's praise. As networks
Re: [WISPA] Full BGP on RouterOS
On 11/02/2010 05:37 PM, Kristian Hoffmann wrote: On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. Yes. pfSense. I'm running that here for dhcp/dns/vpn and terminating VLANs. I would like to know it's performance for full BGP feeds. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
I've never heard of bsdrp before. I'll have to check it out. I was planning on doing a full freebsd install.. I've been using FreeBSD for 10 years and it is my platform of choice. Gerard On Tue, Nov 2, 2010 at 7:52 PM, Scott Lambert lamb...@lambertfam.org wrote: On Tue, Nov 02, 2010 at 05:23:40PM -0400, Gerard Dupont wrote: I have another x86 system that I think I'm going to load freebsd on it and just run Quagga. I don't want to spend the money on a cisco to handle 3+ full bgp feeds and 300+mbit of traffic. bsdrp.org? or a full FreeBSD install? I just configured my first bsdrp install. It's not in production yet, but configuring it didn't suck. I'm a cisco guy so I feel more at home in Quagga. I setup all but the primary LAN IP in quagga rather than rc.conf so that I can be closer to seeing/changing everything from one place. We also have a few older ImageStream Transport routers. The are pretty nice to work with. They won't handle 300Mbps of traffic but the work great at our tower sites. We only have about 150Mbps traffic total anyway. Just realized they even have mtr installed. The management interface is probably better than the BSDRP methods for non-FreeBSD gearheads. I still need to try a Vyatta system. -- Scott Lambert KC5MLE Unix SysAdmin lamb...@lambertfam.org WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
We had problems loading Mikrotik on the supermicro atom 330 dual-core at first. Took some finagling. Something like had to install 5.x on it, and then downgrade it to 4.x from winbox, Its all hazy now.. Routermaxx 1200 is rocksolid (Axiomtek appliance) Nick Olsen Network Operations (855) FLSPEED x106 From: Kristian Hoffmann kh...@fire2wire.com Sent: Tuesday, November 02, 2010 8:36 PM To: WISPA General List wireless@wispa.org Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmann kh...@fire2wire.com To: WISPA General List wireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Tue, Nov 02, 2010 at 06:13:19PM -0700, Charles N Wyble wrote: On 11/02/2010 05:37 PM, Kristian Hoffmann wrote: On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. Yes. pfSense. I'm running that here for dhcp/dns/vpn and terminating VLANs. I would like to know it's performance for full BGP feeds. The openbgpd package in pfSense three months ago wasn't ready for prime time. I tried it just recieving default and advertising 2 networks. I swapped it for one of the Imagestream Transports pretty quickly. It's not good when openbgpd removes the default, (static) route, and openospfd adds it's dynamic route and openbgpd can't override openospfd's route to put the default back in. pfSense is great. The integration with openBGP, not so much. I suspect pkg_adding quagga would be more productive. -- Scott LambertKC5MLE Unix SysAdmin lamb...@lambertfam.org WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Tue, Nov 02, 2010 at 10:18:46PM -0400, Gerard Dupont wrote: I've never heard of bsdrp before. I'll have to check it out. I was planning on doing a full freebsd install.. I've been using FreeBSD for 10 years and it is my platform of choice. bsdrp is nice in that they have a good nanobsd setup for compact flash. It handles updating about as well as pfSense. The project is young, but it seems alright. -- Scott LambertKC5MLE Unix SysAdmin lamb...@lambertfam.org WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
Tom, I agree that Linux works very well as a router, but it still doesn't compare to a dedicated hardware platform (like Cisco) that was built from the ground up to do nothing but routing. We purchased a used Cisco 12008 router about 1.5 years ago off ebay. They are very, very cheap... the only downside is they are BIG and require 240VAC. But it's way cool to pull the CPU card while the router is moving 500Mbps of traffic and have it not even miss a single ping (due to the redundant CPU card). Same goes for the route fabric card. ;) We use Mikrotik for our inside core router and this big Cisco for our border router to our BGP upstreams. I have slept very well for the last 1.5 years knowing everything in the box is fully redundant (CPU, route, power, etc.). :) Travis Microserv On 11/2/2010 9:04 PM, Tom DeReggi wrote: Note: Quagga has been very reliable for quite some time now. Imagestream and Vyatta both use Quagga. Both are great choices for BGP routers. I personally use Mandrake (Mandriva) Linux with a slew of custom modifications that we have made, loaded on SuperMicro, and then use latest Quagga. That has worked well for us, the last 5 years. (although, I dont recommend that to someone, until they are vastly familiar with their distro of Linux. Last thing you want to do is use your BGP router for a Guinee Pig Science project, rebooting it all the time to test script changes.) But once you are comfortable with your Distro, it works well. There are a million arguements for and against Cisco versus Linux, to be used for the ISPs' average NOC/POP router/switch. I dont dispute any of the arguements. But one area where I believe Linux stands tall, is as a CORE BGP router. A core BGP router can be one of the more simplistic configured routers because it only really needs to perform one function, BGP routing to its connected peers. For BGP there are two critical needs Fast processors and Lots of RAM. In todays world there is no excuse to not have both of those. The problem with Cisco is that it lacks both, unless you pay big bucks. Linux on the other hand has an abundance of both, when combined with PC-Like hardware. I laugh at my competitors, when they say, oh no, BGP reset, had to reload BGP tables, now there is latency for like 3 minutes or compromised routing for that period or got a route problem, the small prefixes aren't in my tables. . On Linux, if you want to restart BGP, well thats like 1 second to reload tables. And no need to drop any routes, unless you want to. You could have Full routes with like 30 peers from a single router, if you wanted to. You can load up Linux with like 32 NICs (qty8 4port GIG NICs) in a 2U case, if you want to, and dont even need a Switch. (Although new will cost you about $430 per 4port PCI-E Gig NIC). Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Kristian Hoffmannkh...@fire2wire.com To: WISPA General Listwireless@wispa.org Sent: Tuesday, November 02, 2010 8:37 PM Subject: Re: [WISPA] Full BGP on RouterOS On Tue, 2010-11-02 at 18:52 -0500, Scott Lambert wrote: I still need to try a Vyatta system. I loathe the idea of managing a *nix distro on a router (which is why we use RouterOS now). Apparently I've had too much Tik-aid, because I had completely forgotten about Vyatta and similar options. I have a SuperMicro 5015A-H (Atom 330 dual-core) coming in tomorrow. I'm going to try RouterOS and Vyatta and see how BGP responds on each with a single feed. If anyone else has an x86-based distro they'd like to see performance on, let me know. And thanks for all the responses. The information has been very helpful. Unfortunately, the conclusion I came to is I have no idea what I'm going to do. Cisco = $$$ and MikroTik = coin flip. Hopefully Vyatta lands somewhere in the middle. Thanks, -Kristian WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org
Re: [WISPA] Full BGP on RouterOS
We have full routes on two RB1000, it takes a couple of minuets with high CPU before it finishes loading routes. As soon as there is a path for traffic then it starts flowing without much, if any, delay on traffic. I have not tried printing the routes at the terminal. On Fri, Oct 29, 2010 at 10:51 AM, Kristian Hoffmann kh...@fire2wire.com wrote: Hi, Does anyone have 1-2 full BGP routing tables on a MikroTik router? If so, what kind of hardware are you running. I'm testing a single feed on a P3 800. It loads the routes fine, and seems to handle the routes in stride (all 328659 of them), until you start poking at the routing table like... /ip route print count-only where bgp-as-path=1234 An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Thanks, -- Kristian Hoffmann System Administrator kh...@fire2wire.com http://www.fire2wire.com Office - 209-543-1800 | Fax - 209-545-1469 | Toll Free - 800-905-FIRE WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Full BGP on RouterOS
On Fri, 2010-10-29 at 08:51 -0700, Kristian Hoffmann wrote: An AS that yielded 500 routes took 1-2 minutes at 100% CPU to complete. Is this normal these days, or is significantly greater hardware in order? I used to have a full feed on a Cisco 3640. It took 5-10 minutes to load all of the routes after a reload, and it was almost impossible to log in, high packet loss, etc. during that time. So, should it take 10 seconds on real hardware, or is this type of query always slow? Given the hardware you are using, that is about normal. Upgrading will certainly make this better/faster. It is only the searching that is a problem in most cases, as the router is not likely to have a problem looking up the best path. So, I guess you could call this a cosmetic problem (only when you want to look). More CPU=faster in this case, but only faster for you. -- * Butch Evans * Professional Network Consultation* * http://www.butchevans.com/* Network Engineering * * http://store.wispgear.net/* Wired or Wireless Networks * * http://blog.butchevans.com/ * ImageStream, Mikrotik and MORE! * WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
