See, I'm in a somewhat of a unique boat on this, so I'll give some perspective.
Brand new Fiber ISP right outside KC Metro. 144ct going in everywhere with spare duct. Combo above ground PED w/ below ground slack storage. Redundant ring, half on poles, half buried. Two providers feeding us, with hospitals and school districts excited about turning up 1-10000Mbps service with us over the next couple of months. Netflix and Google peering happening. Two TV content providers getting ready to feed us, plus local TV rebroadcast rights. Upstreams want to skip 40Gbps feeds and move right to 100Gbps when we get past 10Gbps on our feeds (projections show 18-24mo). I needed CGNAT due to the state of IPv4. Also deploying IPv6 along side it. Needed mpls for certain handoffs, and MEF services for the gpon side and other features and classes of service, as well as decent routing table capability and BGP Flowspec. I tried to go with a smaller chassis like that, even tried to get a netiron XMR to work, but I simply couldn't get the feature sets and expandability I needed to make everything work right out of the gate. :( On Jan 15, 2016 9:18 AM, "Eric Kuhnke" <[email protected]> wrote: > > > *The difference with the Junipers and Ciscos and Brocades of the world is > that they often have a higher tier of products for those entities that can > afford it that will support their needs* > > The great thing about buying used/refurb Cisco and Juniper is that there > is a VAST quantity of stuff on the market which is no longer fast or > capable enough for a datacenter/hosting/virtualization/purely fiber ISP > environment, but is real overkill for the average medium sized WISP use. > I'll use an example of a to-remain-nameless adult content hosting > colocation ISP located near a major IX point. They push probably an > aggregate of 120-150Gbps of traffic to their peers and upstreams on a > fairly regular basis and need core routers capable of handling such > volumes, and even more in the future. > > You can buy an identical matched pair of used 7604 or 7606 with RSP720 + > various 10GbE and 1GbE optical interfaces for less than $4500-5000. Such a > configuration for a WISP can handle a port-channeled 2 x 10Gbps link to an > upstream. Is there a WISP on the list that moves more than 12-15Gbps of > traffic? Doing 10GbE with fiber is easy, doing real 1GbE full duplex with > PTP part101 links is *hard and expensive*. The bottleneck in many cases > is the PTP microwave backhaul connection in and out of towers and rooftop > sites, not the router capabilities. It takes a lot of money and a lot of > users to support building a resilient ring of Exalt ExtremeAir 18 GHz 1 > Gbps links serving a county sized area. Huge numbers of WISPs have traffic > patterns to the Internet as a whole that look like 180 to 350Mbps in a > daily cyclic sine wave pattern that fit nicely inside a 1GbE handoff from > their upstream. > > Take that 7604 for example, you can put in a few 24-port SFP linecards > that will do 24 Gbps line rate. And you have a couple of blades each with 4 > x 10GbE. You'd need to be a *big *WISP to outgrow that any time soon. > More likely you'd outgrow the total 1 million routing table entry size > (combined IPV4 + IPV6 FIB) before the forwarding ability of the ASICs. > > > > On Fri, Jan 15, 2016 at 6:52 AM, Josh Reynolds <[email protected]> > wrote: > >> That's not quite true. >> >> For example, on Juniper with a standard switching control board, a >> 16x10Gbps line card has a 4:1 oversub ratio. That interface line card >> retails for something around $160,000 MSRP. With the replacement of the >> standard SCB for an enhanced on, you can get full line rate on all ports. >> >> On an MX240 you might have a 4x10Gbps card, but with firewall acl and >> cgnat enabled you might be limited to 8-9Gbps total throughput through it. >> >> So to say "it just works" on the higher end of vendors and products is >> baloney. The difference with the Junipers and Ciscos and Brocades of the >> world is that they often have a higher tier of products for those entities >> that can afford it that will support their needs. Also their support is >> more highly trained (and there's more of them), and companies will fix your >> shit pretty quick when you buy a few hundred million worth of their >> products. >> On Jan 15, 2016 8:19 AM, "Erich Kaiser" <[email protected]> >> wrote: >> >>> I agree there is a huge price difference between a Mikrotik and a >>> Juniper/Cisco/Brocade router, but you also need to look at the facts. >>> Those routers are made to support a Full routed 10Gbps/40Gbps/100Gbps of >>> throughput per port. With Mikrotik it is a MAYBE it will if I don't do >>> this or do that.. Obviously there are many models out there, but if you >>> look at the true carrier grade router lines, it is a full routed port. >>> >>> I think each one has its place. >>> >>> Just my opinion.. >>> >>> Erich Kaiser >>> North Central Tower >>> [email protected] >>> Office: 630-621-4804 >>> Cell: 630-777-9291 >>> >>> >>> On Fri, Jan 15, 2016 at 7:37 AM, Mike Hammett <[email protected]> wrote: >>> >>>> The #1 thing I learned at NANOG was that the big vendors are just as >>>> full of shit as the little vendors. Cisco, Juniper, AlcaLu, etc. are all >>>> just as likely to ignore problems and create bugs as Mikrotik, Ubiquiti, >>>> etc. >>>> >>>> When sitting at a table at about 1 AM one night with a head engineer >>>> with one international operator and one national operator I said something >>>> to the effect of, "I've just largely been listening to you guys comparing >>>> notes on different products and issues you've had. I use Mikrotik routers >>>> in my network and people often say they are junk and that Cisco, Juniper, >>>> etc. are the way to go. Those vendors are exactly who you guys use and yet >>>> here you are with the same type of complaints as we have with our vendors." >>>> The one at the international vendor chimed in, "and we pay 10x more for >>>> that right." >>>> >>>> Don't lose sight of that. >>>> >>>> For example, Cisco is known for having Ethernet chipset incompatibility >>>> issues.... which is exactly what this sub-topic has been about. >>>> >>>> >>>> To think otherwise is simply blind arrogance. >>>> >>>> >>>> >>>> ----- >>>> Mike Hammett >>>> Intelligent Computing Solutions >>>> http://www.ics-il.com >>>> >>>> <https://www.facebook.com/ICSIL> >>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> >>>> <https://www.linkedin.com/company/intelligent-computing-solutions> >>>> <https://twitter.com/ICSIL> >>>> >>>> Midwest Internet Exchange >>>> http://www.midwest-ix.com >>>> >>>> <https://www.facebook.com/mdwestix> >>>> <https://www.linkedin.com/company/midwest-internet-exchange> >>>> <https://twitter.com/mdwestix> >>>> ------------------------------ >>>> *From: *"Eric Kuhnke" <[email protected]> >>>> *To: *[email protected] >>>> *Sent: *Thursday, January 14, 2016 10:49:12 PM >>>> *Subject: *Re: [AFMUG] Weird Problem with download >>>> >>>> This is sort of an illustration of why I am still very wary of Mikrotik >>>> for routers. >>>> >>>> There's a myriad of Cisco routers available with 12 to 20 SFP-based GbE >>>> interfacess (used/refurb) that are just as capable, if somewhat bigger and >>>> hotter, that run very stable versions of IOS. It's totally unacceptable for >>>> Cisco to break basic functionality for $HUGE_ISP and >>>> $HUGE_ENTERPRISE_CUSTOMER , which are usually using the same IOS trains I >>>> use for service provider functions. >>>> >>>> Mikrotik OS software support is pretty much "hey go over to this forum >>>> and hope somebody answers your question", if something is broken you are >>>> just supposed to wait for a new version release in a month or two. If >>>> something is fundamentally broken in a version of IOS used by a mature (6+ >>>> year old) Cisco platform it's almost certain that somebody much bigger and >>>> more important than yourself has already run into the issue, escalated it >>>> with Cisco, and a new build fixing the bug is available. >>>> >>>> If the space and power are available I would much rather have a 6503E >>>> with dual supervisor than a CCR.... >>>> >>>> On Wed, Jan 13, 2016 at 7:34 PM, Rory Conaway <[email protected]> >>>> wrote: >>>> >>>>> This just gets even better. Worked 24 hours, then breaks again. Up >>>>> to Level 4 tech support. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected]] *On Behalf Of *Rory Conaway >>>>> *Sent:* Wednesday, January 13, 2016 7:56 AM >>>>> >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> We are taking it to the Mikrotik forum. This makes no sense since we >>>>> have 12 of the already running with no issues. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected] <[email protected]>] *On >>>>> Behalf Of *Glen Waldrop >>>>> *Sent:* Wednesday, January 13, 2016 7:54 AM >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> I’m not terribly happy with ROS since 6.29. >>>>> >>>>> >>>>> >>>>> Maybe it’s just still not 100% yet. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* Rory Conaway <[email protected]> >>>>> >>>>> *Sent:* Tuesday, January 12, 2016 8:36 PM >>>>> >>>>> *To:* [email protected] >>>>> >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> Replaced the Mikrotik routers with an Airrouter, it works. Replaced >>>>> with an Edgerouter, still works. This goes under the category of WTF. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected] <[email protected]>] *On >>>>> Behalf Of *Rory Conaway >>>>> *Sent:* Monday, January 11, 2016 9:13 PM >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> We tried 3 Mikrotik routers, 2 cable modems, and two radios from two >>>>> different manufacturers. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected] <[email protected]>] *On >>>>> Behalf Of *Glen Waldrop >>>>> *Sent:* Monday, January 11, 2016 8:52 PM >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> MTU? >>>>> >>>>> Last time I had that problem was an inconsistent Ethernet connection. >>>>> It wasn’t showing errors, but UDP was dropping packets. TCP retransmitted. >>>>> >>>>> UDP was screwy, but TCP was downloading at the limit of the wireless >>>>> feed. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* Rory Conaway <[email protected]> >>>>> >>>>> *Sent:* Monday, January 11, 2016 9:47 PM >>>>> >>>>> *To:* [email protected] >>>>> >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> Nailed the problem down to the Mikrotik router having a problem with >>>>> DNS caching whether it’s Cox or Google. Not sure where to go from there >>>>> since the other 12 don’t have a problem. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected] <[email protected]>] *On >>>>> Behalf Of *Rory Conaway >>>>> *Sent:* Monday, January 11, 2016 4:44 PM >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> We had raons bit no dangling cables. Level 3 says no errors to the >>>>> modem and we had a tech out already. We are changing the 2011 to a 450 >>>>> just to confirm before we eacalate further. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Rory Conaway >>>>> >>>>> Triad Wireless >>>>> >>>>> >>>>> >>>>> Typed on my phone with one finger so please excuse typos and >>>>> abbreviations. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -------- Original message -------- >>>>> >>>>> From: That One Guy /sarcasm <[email protected]> >>>>> >>>>> Date: 1/11/2016 10:38 AM (GMT-07:00) >>>>> >>>>> To: [email protected] >>>>> >>>>> Subject: Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> If it just started and youre in an area hit by storms, some of the >>>>> neighbors, mostly the empty houses with no tenants at this point, cable >>>>> was >>>>> busted down by tree limbs and is dangling terminated byt the mud now >>>>> >>>>> >>>>> >>>>> On Mon, Jan 11, 2016 at 10:58 AM, Glen Waldrop <[email protected]> >>>>> wrote: >>>>> >>>>> Likely an RF issue on the cable side. >>>>> >>>>> I see this sort of thing with the company I consult for. They’ll have >>>>> me going through the router, trying to shut down the “virus” or peer to >>>>> peer when it is thousands and thousands of retransmits and broken packets. >>>>> >>>>> >>>>> >>>>> Just like our wireless it can have good quality until loaded, then >>>>> watch the quality drop to nearly nothing and loose connection. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* Rory Conaway <[email protected]> >>>>> >>>>> *Sent:* Sunday, January 10, 2016 8:25 PM >>>>> >>>>> *To:* [email protected] >>>>> >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> The Netgear router has nothing to do with it, I was just using their >>>>> website as an example of downloading a file. It’s computer, 2011, cable >>>>> modem, the world. I just can’t download anything from any sites. What’s >>>>> weird is they just drop. >>>>> >>>>> >>>>> >>>>> Rory >>>>> >>>>> >>>>> >>>>> *From:* Af [mailto:[email protected]] *On Behalf Of *Ken Hohhof >>>>> *Sent:* Sunday, January 10, 2016 5:12 PM >>>>> *To:* [email protected] >>>>> *Subject:* Re: [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> You mention Netgear, is this a Netgear router? They used to have a >>>>> problem with downloads stalling, but that was quite a few years ago. The >>>>> fix was to disable SPI Firewall. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* Rory Conaway <[email protected]> >>>>> >>>>> *Sent:* Sunday, January 10, 2016 6:06 PM >>>>> >>>>> *To:* [email protected] >>>>> >>>>> *Subject:* [AFMUG] Weird Problem with download >>>>> >>>>> >>>>> >>>>> We have a cable circuit that can do everything except download. There >>>>> is a Mikrotik router between the computer and the cable modem. Haven’t >>>>> tested taking the router out yet but we have tried 2 routers, same >>>>> results. Basically, downloads either don’t start or start and then just >>>>> crash. Game machines can’t update, even downloading a 20MB file from >>>>> netgear fails. Everything else seems to work. Any ideas would be >>>>> helpful. >>>>> >>>>> >>>>> >>>>> *Rory Conaway **• Triad Wireless •** CEO* >>>>> >>>>> *4226 S. 37th Street • Phoenix • AZ 85040* >>>>> >>>>> *602-426-0542 <602-426-0542>* >>>>> >>>>> *[email protected] <[email protected]>* >>>>> >>>>> *www.triadwireless.net <http://www.triadwireless.net/>* >>>>> >>>>> >>>>> >>>>> “Life is full of unfair calls, missed plays, and bad catches. But >>>>> true baseball players keep on playing” - Lessons from Baseball >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> If you only see yourself as part of the team but you don't see your >>>>> team as part of yourself you have already failed as part of the team. >>>>> >>>> >>>> >>>> >>> >
