Thanks all, this is a very useful discussion. And I'm particularly encouraged by Paul Stewart's comments!
As Eric Kuhnke remarked, by flat I meant all routers in OSPF area 0. We do use a single layer two network with multiple vlans within each apartment building or condo complex, but it's all routed between buildings or building complexes, usually with at least two paths to each complex. The network I described is the largest of several fiber-fed, radio-interconnected networks that have private ASNs, iBGP in between and BGP confederation so each network's default route is upstream on its fiber but any network can failover via radio during a fiber outage. In the largest network (the one with 600+ routers) the only measurable issue I can see today is the time for routes to propagate. If I add a route or I throw traffic in one part of the network onto a backup path, the route change can take 2-3 seconds to appear on a router that is far removed from where the change happened. But that's the only issue I can see during normal operation. We have had occasions (luckily not very often) when someone has introduce an address conflict. Besides screwing up a specific existing customer, this typically causes a route flap which is visible in Winbox looking at IP, Routes - the relevant portion of the table jumps up and down by one route at a rate of about 2-3 seconds. Thanks, Brough Brough Turner netBlazr Inc. – Free your Broadband! Mobile: 617-285-0433 Skype: brough netBlazr Inc. <http://www.netblazr.com/> | Google+ <https://plus.google.com/102447512447094746687/posts?hl=en> | Twitter <https://twitter.com/#%21/brough> | LinkedIn <http://www.linkedin.com/in/broughturner> | Facebook <http://www.facebook.com/brough.turner> | Blog <http://blogs.broughturner.com/> | Personal website <http://broughturner.com/> On Fri, Jun 8, 2018 at 3:16 PM, Paul Stewart <[email protected]> wrote: > I’ve done a lot of work with that model in the past and it works very well > …. Extremely flexible what you can then do for multi-service handoff as > well 😉 > > > > -p > > > > *From: *Af <[email protected]> on behalf of Carl Peterson < > [email protected]> > *Reply-To: *<[email protected]> > *Date: *Friday, June 8, 2018 at 1:52 PM > > *To: *<[email protected]> > *Subject: *Re: [AFMUG] OSPF - How large can a flat network grow? > > > > I've been thinking a lot about the flat -> routed transition lately and my > current thinking is that we over reacted and just went to routers > everywhere. We have been trimming back a lot and moving to running QinQ > over VPLS where every sub has their own CVLAN in an SVLAN. As I get more > comfortable with it, I'm thinking about expanding it and dropping more of > the "core" locations with routers. Instead of A--B--C all with routers, we > would drop the router at B and just run B with a primary and secondary VPLS > circuit, say primary through A and secondary through C. Saves a ton on > enclosures, batteries etc. > > > > On Fri, Jun 8, 2018 at 12:54 PM, Paul Stewart <[email protected]> > wrote: > > Can’t comment on “small iron” routers as limited experience but can tell > you with larger gear (which in theory has a lot more CPU/processing > capability) that large OSPF networks (100k routes) exist and work just > fine. One company I consulted for a number of years ago had over 600k > routes in OSPF .. seriously … and yes they did experience some issues but > they were not big enough issues to warrant changing til a few years later > when they migrated everything to ISIS anyways. That’s the most extreme > example and not one I personally recommend 😉 > > > > As others have mentioned, it’s a hard question to answer as there is no > “one size fits all” … often it’s more about how the network is designed > then specific sizes of routes or numbers of routers… > > > > Paul > > > > > > *From: *Af <[email protected]> on behalf of Eric Kuhnke < > [email protected]> > *Reply-To: *<[email protected]> > *Date: *Thursday, June 7, 2018 at 6:06 PM > *To: *<[email protected]> > *Subject: *Re: [AFMUG] OSPF - How large can a flat network grow? > > > > Also worth mentioning that a lot of OSPF documentation available on the > Internet, makes assumptions that were valid in 2002 or so... When a > typical router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or > even something smaller like a 2621. > > > > Probably still true if you're trying to do OSPF on very small Mikrotiks > but not as much of a concern in the modern era. The main bottleneck in > routing platforms is FIB size and RAM for BGP tables, not so much OSPF. > > > > > > > > On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess <[email protected]> > wrote: > > This is a “unanswerable” question. In honesty, you can have 20k routes in > OSPF and it be responsive so the routing platform does not have a limit. > The number of routers, is another issue, if you have 500 routers all fiber > connected and they don’t go up/down much, then no big deal, however, if > they do go up down quite a bit or you don’t know how to manage them, then > yes this can be a factor. The last thing is convergence time, if you have > lots of fiber, and/or well connected routers, then that is not a major > issue, break part of your network and see how long it takes for a reroute, > if that is acceptable, then again no worries. > > > > Now OSPF books, state that you should have no more than 75-100 routers, > but I have read things that state no more than 50 and I have other networks > that have more than 500. So.. Again, it’s not a good answerable question. > > > > However, my suggestion is to look at your network as a whole and see if > there is some kind of logic, to splitting up your OSPF domains. You can > use OSPF areas, or you can use BGP between them. But there needs to be a > good, constant method to splitting your network like that. Keep in mind > that using defaults will cause traffic to shift, etc, so you need to plan > plan plan… > > > > Just my two cents. > > > > > > > > *Dennis Burgess, Mikrotik Certified Trainer * > > Author of "Learn RouterOS- Second Edition” > > *Link Technologies, Inc* -- Mikrotik & WISP Support Services > > *Office*: 314-735-0270 Website: http://www.linktechs.net > > Create Wireless Coverage’s with www.towercoverage.com > > > > *From:* Af <[email protected]> *On Behalf Of *Brough Turner > *Sent:* Thursday, June 7, 2018 1:41 PM > *To:* [email protected] > *Subject:* [AFMUG] OSPF - How large can a flat network grow? > > > > We're an urban WISP with a dense mesh of wireless links and a router per > building. I am concerned that, without paying attention, we have grown to > 600+ routers and ~2550 routes in one OSPF domain. This network has a > diverse mix of routers from CCR1036s down to RB750UPs. We're not having any > OSPF problems at this time and I have plenty of other things to worry > about, but I'd hate to hit some limit and have the whole thing blow up. > > Does anyone have experience (positive or negative) with large flat OSPF > networks? > And, if you have had problems, what were the problems? > > Thanks, > Brough > > Brough Turner > netBlazr Inc. – Free your Broadband! > Mobile: 617-285-0433 Skype: brough > netBlazr Inc. <http://www.netblazr.com/> | Google+ > <https://plus.google.com/102447512447094746687/posts?hl=en> | Twitter > <https://twitter.com/#%21/brough> | LinkedIn > <http://www.linkedin.com/in/broughturner> | Facebook > <http://www.facebook.com/brough.turner> | Blog > <http://blogs.broughturner.com/> | Personal website > <http://broughturner.com/> > > > > > > > > > > -- > > Carl Peterson > > *PORT NETWORKS* > > 401 E Pratt St, Ste 2553 > <https://maps.google.com/?q=401+E+Pratt+St,+Ste+2553+Baltimore,+MD+21202&entry=gmail&source=g> > > Baltimore, MD 21202 > <https://maps.google.com/?q=401+E+Pratt+St,+Ste+2553+Baltimore,+MD+21202&entry=gmail&source=g> > > (410) 637-3707 >
