Re: [AFMUG] OSPF - How large can a flat network grow?

[Brough Turner]

Thanks all, this is a very useful discussion.
And I'm particularly encouraged by Paul Stewart's comments!

As Eric Kuhnke remarked, by flat I meant all routers in OSPF area 0. We do
use a single layer two network with multiple vlans within each apartment
building or condo complex, but it's all routed between buildings or
building complexes, usually with at least two paths to each complex. The
network I described is the largest of several fiber-fed,
radio-interconnected networks that have private ASNs, iBGP in between and
BGP confederation so each network's default route is upstream on its fiber
but any network can failover via radio during a fiber outage.

In the largest network (the one with 600+ routers) the only measurable
issue I can see today is the time for routes to propagate. If I add a route
or I throw traffic in one part of the network onto a backup path, the route
change can take 2-3 seconds to appear on a router that is far removed from
where the change happened. But that's the only issue I can see during
normal operation.

We have had occasions (luckily not very often) when someone has introduce
an address conflict.  Besides screwing up a specific existing customer,
this typically causes a route flap which is visible in Winbox looking at
IP, Routes - the relevant portion of the table jumps up and down by one
route at a rate of about 2-3 seconds.

Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc. <http://www.netblazr.com/> | Google+
<https://plus.google.com/102447512447094746687/posts?hl=en> | Twitter
<https://twitter.com/#%21/brough> | LinkedIn
<http://www.linkedin.com/in/broughturner> | Facebook
<http://www.facebook.com/brough.turner> | Blog
<http://blogs.broughturner.com/> | Personal website
<http://broughturner.com/>



On Fri, Jun 8, 2018 at 3:16 PM, Paul Stewart  wrote:

> I’ve done a lot of work with that model in the past and it works very well
> …. Extremely flexible what you can then do for multi-service handoff as
> well 
>
>
>
> -p
>
>
>
> *From: *Af  on behalf of Carl Peterson <
> cpeter...@portnetworks.com>
> *Reply-To: *
> *Date: *Friday, June 8, 2018 at 1:52 PM
>
> *To: *
> *Subject: *Re: [AFMUG] OSPF - How large can a flat network grow?
>
>
>
> I've been thinking a lot about the flat -> routed transition lately and my
> current thinking is that we over reacted and just went to routers
> everywhere.  We have been trimming back a lot and moving to running QinQ
> over VPLS where every sub has their own CVLAN in an SVLAN.  As I get more
> comfortable with it, I'm thinking about expanding it and dropping more of
> the "core" locations with routers.  Instead of A--B--C all with routers, we
> would drop the router at B and just run B with a primary and secondary VPLS
> circuit, say primary through A and secondary through C.  Saves a ton on
> enclosures, batteries etc.
>
>
>
> On Fri, Jun 8, 2018 at 12:54 PM, Paul Stewart 
> wrote:
>
> Can’t comment on “small iron” routers as limited experience but can tell
> you with larger gear (which in theory has a lot more CPU/processing
> capability) that large OSPF networks (100k routes) exist and work just
> fine.  One company I consulted for a number of years ago had over 600k
> routes in OSPF .. seriously … and yes they did experience some issues but
> they were not big enough issues to warrant changing til a few years later
> when they migrated everything to ISIS anyways.  That’s the most extreme
> example and not one I personally recommend 
>
>
>
> As others have mentioned, it’s a hard question to answer as there is no
> “one size fits all” … often it’s more about how the network is designed
> then specific sizes of routes or numbers of routers…
>
>
>
> Paul
>
>
>
>
>
> *From: *Af  on behalf of Eric Kuhnke <
> eric.kuh...@gmail.com>
> *Reply-To: *
> *Date: *Thursday, June 7, 2018 at 6:06 PM
> *To: *
> *Subject: *Re: [AFMUG] OSPF - How large can a flat network grow?
>
>
>
> Also worth mentioning that a lot of OSPF documentation available on the
> Internet, makes assumptions that were valid in 2002 or so...  When a
> typical router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or
> even something smaller like a 2621.
>
>
>
> Probably still true if you're trying to do OSPF on very small Mikrotiks
> but not as much of a concern in the modern era. The main bottleneck in
> routing platforms is FIB size and RAM for BGP tables, not so much OSPF.
>
>
>
>
>
>
>
> On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess 
> wrote:
>
> This is a “unanswerable” question.  In honesty, you can have 20k routes in
> OSPF and it be responsive so the routing 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Paul Stewart]

I’ve done a lot of work with that model in the past and it works very well …. 
Extremely flexible what you can then do for multi-service handoff as well 

 

-p

 

From: Af  on behalf of Carl Peterson 

Reply-To: 
Date: Friday, June 8, 2018 at 1:52 PM
To: 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

 

I've been thinking a lot about the flat -> routed transition lately and my 
current thinking is that we over reacted and just went to routers everywhere.  
We have been trimming back a lot and moving to running QinQ over VPLS where 
every sub has their own CVLAN in an SVLAN.  As I get more comfortable with it, 
I'm thinking about expanding it and dropping more of the "core" locations with 
routers.  Instead of A--B--C all with routers, we would drop the router at B 
and just run B with a primary and secondary VPLS circuit, say primary through A 
and secondary through C.  Saves a ton on enclosures, batteries etc.  

 

On Fri, Jun 8, 2018 at 12:54 PM, Paul Stewart  wrote:

Can’t comment on “small iron” routers as limited experience but can tell you 
with larger gear (which in theory has a lot more CPU/processing capability) 
that large OSPF networks (100k routes) exist and work just fine.  One company I 
consulted for a number of years ago had over 600k routes in OSPF .. seriously … 
and yes they did experience some issues but they were not big enough issues to 
warrant changing til a few years later when they migrated everything to ISIS 
anyways.  That’s the most extreme example and not one I personally recommend 

 

As others have mentioned, it’s a hard question to answer as there is no “one 
size fits all” … often it’s more about how the network is designed then 
specific sizes of routes or numbers of routers… 

 

Paul

 

 

From: Af  on behalf of Eric Kuhnke 
Reply-To: 
Date: Thursday, June 7, 2018 at 6:06 PM
To: 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

 

Also worth mentioning that a lot of OSPF documentation available on the 
Internet, makes assumptions that were valid in 2002 or so...  When a typical 
router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or even something 
smaller like a 2621. 

 

Probably still true if you're trying to do OSPF on very small Mikrotiks but not 
as much of a concern in the modern era. The main bottleneck in routing 
platforms is FIB size and RAM for BGP tables, not so much OSPF. 

 

 

 

On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess  wrote:

This is a “unanswerable” question.  In honesty, you can have 20k routes in OSPF 
and it be responsive so the routing platform does not have a limit.  The number 
of routers, is another issue, if you have 500 routers all fiber connected and 
they don’t go up/down much, then no big deal, however, if they do go up down 
quite a bit or you don’t know how to manage them, then yes this can be a 
factor. The last thing is convergence time, if you have lots of fiber, and/or 
well connected routers, then that is not a major issue, break part of your 
network and see how long it takes for a reroute, if that is acceptable, then 
again no worries.

 

Now OSPF books, state that you should have no more than 75-100 routers, but I 
have read things that state no more than 50 and I have other networks that have 
more than 500. So..  Again, it’s not a good answerable question.  

 

However, my suggestion is to look at your network as a whole and see if there 
is some kind of logic, to splitting up your OSPF domains.  You can use OSPF 
areas, or you can use BGP between them.  But there needs to be a good, constant 
method to splitting your network like that.  Keep in mind that using defaults 
will cause traffic to shift, etc, so you need to plan plan plan…

 

Just my two cents.

 

 

 

Dennis Burgess, Mikrotik Certified Trainer 

Author of "Learn RouterOS- Second Edition” 

Link Technologies, Inc -- Mikrotik & WISP Support Services 

Office: 314-735-0270  Website: http://www.linktechs.net 

Create Wireless Coverage’s with www.towercoverage.com 

 

From: Af  On Behalf Of Brough Turner
Sent: Thursday, June 7, 2018 1:41 PM
To: af@afmug.com
Subject: [AFMUG] OSPF - How large can a flat network grow?

 

We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat OSPF 
networks?
And, if you have had problems, what were the problems? 
Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc. | Google+ | Twitter | LinkedIn | Facebook | Blog | Personal 
website 

 

 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Carl Peterson]

I've been thinking a lot about the flat -> routed transition lately and my
current thinking is that we over reacted and just went to routers
everywhere.  We have been trimming back a lot and moving to running QinQ
over VPLS where every sub has their own CVLAN in an SVLAN.  As I get more
comfortable with it, I'm thinking about expanding it and dropping more of
the "core" locations with routers.  Instead of A--B--C all with routers, we
would drop the router at B and just run B with a primary and secondary VPLS
circuit, say primary through A and secondary through C.  Saves a ton on
enclosures, batteries etc.

On Fri, Jun 8, 2018 at 12:54 PM, Paul Stewart  wrote:

> Can’t comment on “small iron” routers as limited experience but can tell
> you with larger gear (which in theory has a lot more CPU/processing
> capability) that large OSPF networks (100k routes) exist and work just
> fine.  One company I consulted for a number of years ago had over 600k
> routes in OSPF .. seriously … and yes they did experience some issues but
> they were not big enough issues to warrant changing til a few years later
> when they migrated everything to ISIS anyways.  That’s the most extreme
> example and not one I personally recommend 
>
>
>
> As others have mentioned, it’s a hard question to answer as there is no
> “one size fits all” … often it’s more about how the network is designed
> then specific sizes of routes or numbers of routers…
>
>
>
> Paul
>
>
>
>
>
> *From: *Af  on behalf of Eric Kuhnke <
> eric.kuh...@gmail.com>
> *Reply-To: *
> *Date: *Thursday, June 7, 2018 at 6:06 PM
> *To: *
> *Subject: *Re: [AFMUG] OSPF - How large can a flat network grow?
>
>
>
> Also worth mentioning that a lot of OSPF documentation available on the
> Internet, makes assumptions that were valid in 2002 or so...  When a
> typical router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or
> even something smaller like a 2621.
>
>
>
> Probably still true if you're trying to do OSPF on very small Mikrotiks
> but not as much of a concern in the modern era. The main bottleneck in
> routing platforms is FIB size and RAM for BGP tables, not so much OSPF.
>
>
>
>
>
>
>
> On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess 
> wrote:
>
> This is a “unanswerable” question.  In honesty, you can have 20k routes in
> OSPF and it be responsive so the routing platform does not have a limit.
> The number of routers, is another issue, if you have 500 routers all fiber
> connected and they don’t go up/down much, then no big deal, however, if
> they do go up down quite a bit or you don’t know how to manage them, then
> yes this can be a factor. The last thing is convergence time, if you have
> lots of fiber, and/or well connected routers, then that is not a major
> issue, break part of your network and see how long it takes for a reroute,
> if that is acceptable, then again no worries.
>
>
>
> Now OSPF books, state that you should have no more than 75-100 routers,
> but I have read things that state no more than 50 and I have other networks
> that have more than 500. So..  Again, it’s not a good answerable question.
>
>
>
> However, my suggestion is to look at your network as a whole and see if
> there is some kind of logic, to splitting up your OSPF domains.  You can
> use OSPF areas, or you can use BGP between them.  But there needs to be a
> good, constant method to splitting your network like that.  Keep in mind
> that using defaults will cause traffic to shift, etc, so you need to plan
> plan plan…
>
>
>
> Just my two cents.
>
>
>
>
>
>
>
> *Dennis Burgess, Mikrotik Certified Trainer *
>
> Author of "Learn RouterOS- Second Edition”
>
> *Link Technologies, Inc* -- Mikrotik & WISP Support Services
>
> *Office*: 314-735-0270  Website: http://www.linktechs.net
>
> Create Wireless Coverage’s with www.towercoverage.com
>
>
>
> *From:* Af  *On Behalf Of *Brough Turner
> *Sent:* Thursday, June 7, 2018 1:41 PM
> *To:* af@afmug.com
> *Subject:* [AFMUG] OSPF - How large can a flat network grow?
>
>
>
> We're an urban WISP with a dense mesh of wireless links and a router per
> building. I am concerned that, without paying attention, we have grown to
> 600+ routers and ~2550 routes in one OSPF domain. This network has a
> diverse mix of routers from CCR1036s down to RB750UPs. We're not having any
> OSPF problems at this time and I have plenty of other things to worry
> about, but I'd hate to hit some limit and have the whole thing blow up.
>
> Does anyone have experience (positive or negative) with large flat OSPF
> networks?
> And, if you have had problems, what were the problems?
>
> 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Paul Stewart]

Can’t comment on “small iron” routers as limited experience but can tell you 
with larger gear (which in theory has a lot more CPU/processing capability) 
that large OSPF networks (100k routes) exist and work just fine.  One company I 
consulted for a number of years ago had over 600k routes in OSPF .. seriously … 
and yes they did experience some issues but they were not big enough issues to 
warrant changing til a few years later when they migrated everything to ISIS 
anyways.  That’s the most extreme example and not one I personally recommend 

 

As others have mentioned, it’s a hard question to answer as there is no “one 
size fits all” … often it’s more about how the network is designed then 
specific sizes of routes or numbers of routers… 

 

Paul

 

 

From: Af  on behalf of Eric Kuhnke 
Reply-To: 
Date: Thursday, June 7, 2018 at 6:06 PM
To: 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

 

Also worth mentioning that a lot of OSPF documentation available on the 
Internet, makes assumptions that were valid in 2002 or so...  When a typical 
router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or even something 
smaller like a 2621. 

 

Probably still true if you're trying to do OSPF on very small Mikrotiks but not 
as much of a concern in the modern era. The main bottleneck in routing 
platforms is FIB size and RAM for BGP tables, not so much OSPF. 

 

 

 

On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess  wrote:

This is a “unanswerable” question.  In honesty, you can have 20k routes in OSPF 
and it be responsive so the routing platform does not have a limit.  The number 
of routers, is another issue, if you have 500 routers all fiber connected and 
they don’t go up/down much, then no big deal, however, if they do go up down 
quite a bit or you don’t know how to manage them, then yes this can be a 
factor. The last thing is convergence time, if you have lots of fiber, and/or 
well connected routers, then that is not a major issue, break part of your 
network and see how long it takes for a reroute, if that is acceptable, then 
again no worries.

 

Now OSPF books, state that you should have no more than 75-100 routers, but I 
have read things that state no more than 50 and I have other networks that have 
more than 500. So..  Again, it’s not a good answerable question.  

 

However, my suggestion is to look at your network as a whole and see if there 
is some kind of logic, to splitting up your OSPF domains.  You can use OSPF 
areas, or you can use BGP between them.  But there needs to be a good, constant 
method to splitting your network like that.  Keep in mind that using defaults 
will cause traffic to shift, etc, so you need to plan plan plan…

 

Just my two cents.

 

 

 

Dennis Burgess, Mikrotik Certified Trainer 

Author of "Learn RouterOS- Second Edition” 

Link Technologies, Inc -- Mikrotik & WISP Support Services 

Office: 314-735-0270  Website: http://www.linktechs.net 

Create Wireless Coverage’s with www.towercoverage.com 

 

From: Af  On Behalf Of Brough Turner
Sent: Thursday, June 7, 2018 1:41 PM
To: af@afmug.com
Subject: [AFMUG] OSPF - How large can a flat network grow?

 

We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat OSPF 
networks?
And, if you have had problems, what were the problems? 
Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc. | Google+ | Twitter | LinkedIn | Facebook | Blog | Personal 
website 

 

 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Bill Prince]

I think we all started with really flat networks, and I think of them 
the same way as Chuck; all L2 accessible. One subnet. That really only 
works for about 200-250 subscribers, and then you are done.


If you've got 10, 100, or 1000 routes in an OSPF design, you are not 
really "flat" except by OSPF standards, you might be in the same area. 
There's still a lot of routing going on.



bp


On 6/7/2018 5:15 PM, Chuck McCown wrote:

To me, flat network means all MACs are in the same broadcast domain.
That is what I did with my first Canopy system.
Did not even know what a VLAN was back then.
*From:* Gino A. Villarini
*Sent:* Thursday, June 07, 2018 5:47 PM
*To:* af@afmug.com
*Subject:* Re: [AFMUG] OSPF - How large can a flat network grow?
Our network by all means is flat as all services converge into our 
core.� We started by using vlans and qinq but later migrated to mpls/vpls.

Now watching evpn as the next step..
From: Af  on behalf of Lewis Bergman 


Reply-To: "af@afmug.com" 
Date: Thursday, June 7, 2018 at 3:09 PM
To: "af@afmug.com" 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?
Gino would probably be your best source for advice as I'll bet he has 
one of the largest networks of similar construction. I think he went 
MPLS to deal with a variety of issues he was having, including OSPF. 
But maybe Gino can speak up on the issue.
Typically, if you didn't want to do anything else, you would consider 
iBGP and break the OSPF domains up in some logical way. The big 
determining factor for me for such things is the occurrence route 
flapping and how often it happens. Route flapping will be the big 
indicator that you are to big and OSPF can't keep up. You can do some 
tweaking but at some point it just all falls apart. The bad part about 
waiting till that happens is you will loose a lot of customers trying 
to make things stable again.
I would think Denis also would have a much better informed path to 
take and I'll bet he would be happy to contract to help you. Probably 
money well spent.


*//*

*/Gino A. Villarini/*

President
Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968

On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:

We're an urban WISP with a dense mesh of wireless links and a
router per building. I am concerned that, without paying
attention, we have grown to 600+ routers and ~2550 routes in one
OSPF domain. This network has a diverse mix of routers from
CCR1036s down to RB750UPs. We're not having any OSPF problems at
this time and I have plenty of other things to worry about, but
I'd hate to hit some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat
OSPF networks?
And, if you have had problems, what were the problems?

Thanks,
Brough

Brough Turner
netBlazr Inc. � Free your Broadband!
Mobile: 617-285-0433  Skype:� brough
netBlazr Inc. <http://www.netblazr.com/>| Google+
<https://plus.google.com/102447512447094746687/posts?hl=en> |
Twitter <https://twitter.com/#%21/brough> | LinkedIn
<http://www.linkedin.com/in/broughturner> | Facebook
<http://www.facebook.com/brough.turner> | Blog
<http://blogs.broughturner.com/> | Personal website
<http://broughturner.com/>

Re: [AFMUG] OSPF - How large can a flat network grow?

[Chuck McCown]

Yeah, nightmare...

From: Eric Kuhnke 
Sent: Thursday, June 07, 2018 6:33 PM
To: af@afmug.com 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

He's talking about having all routers in OSPF area 0, not a truly flat L2 
broadcast-domain nightmare. 


On Thu, Jun 7, 2018 at 5:15 PM, Chuck McCown  wrote:

  To me, flat network means all MACs are in the same broadcast domain.  
  That is what I did with my first Canopy system.  
  Did not even know what a VLAN was back then.  

  From: Gino A. Villarini 
  Sent: Thursday, June 07, 2018 5:47 PM
  To: af@afmug.com 
  Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

  Our network by all means is flat as all services converge into our core.  We 
started by using vlans and qinq but later migrated to mpls/vpls.

  Now watching evpn as the next step..

  From: Af  on behalf of Lewis Bergman 

  Reply-To: "af@afmug.com" 
  Date: Thursday, June 7, 2018 at 3:09 PM
  To: "af@afmug.com" 
  Subject: Re: [AFMUG] OSPF - How large can a flat network grow?


  Gino would probably be your best source for advice as I'll bet he has one of 
the largest networks of similar construction. I think he went MPLS to deal with 
a variety of issues he was having, including OSPF. But maybe Gino can speak up 
on the issue.  
  Typically, if you didn't want to do anything else, you would consider iBGP 
and break the OSPF domains up in some logical way. The big determining factor 
for me for such things is the occurrence route flapping and how often it 
happens. Route flapping will be the big indicator that you are to big and OSPF 
can't keep up. You can do some tweaking but at some point it just all falls 
apart. The bad part about waiting till that happens is you will loose a lot of 
customers trying to make things stable again.
  I would think Denis also would have a much better informed path to take and 
I'll bet he would be happy to contract to help you. Probably money well spent.




Gino A. Villarini
   
President 
Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968 



  On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:

We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat OSPF 
networks?
And, if you have had problems, what were the problems?

Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc. | Google+ | Twitter | LinkedIn | Facebook | Blog | Personal 
website 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Eric Kuhnke]

He's talking about having all routers in OSPF area 0, not a truly flat L2
broadcast-domain nightmare.


On Thu, Jun 7, 2018 at 5:15 PM, Chuck McCown  wrote:

> To me, flat network means all MACs are in the same broadcast domain.
> That is what I did with my first Canopy system.
> Did not even know what a VLAN was back then.
>
> *From:* Gino A. Villarini
> *Sent:* Thursday, June 07, 2018 5:47 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] OSPF - How large can a flat network grow?
>
> Our network by all means is flat as all services converge into our core.
> We started by using vlans and qinq but later migrated to mpls/vpls.
>
> Now watching evpn as the next step..
>
> From: Af  on behalf of Lewis Bergman <
> lewis.berg...@gmail.com>
> Reply-To: "af@afmug.com" 
> Date: Thursday, June 7, 2018 at 3:09 PM
> To: "af@afmug.com" 
> Subject: Re: [AFMUG] OSPF - How large can a flat network grow?
>
> Gino would probably be your best source for advice as I'll bet he has one
> of the largest networks of similar construction. I think he went MPLS to
> deal with a variety of issues he was having, including OSPF. But maybe Gino
> can speak up on the issue.
> Typically, if you didn't want to do anything else, you would consider iBGP
> and break the OSPF domains up in some logical way. The big determining
> factor for me for such things is the occurrence route flapping and how
> often it happens. Route flapping will be the big indicator that you are to
> big and OSPF can't keep up. You can do some tweaking but at some point it
> just all falls apart. The bad part about waiting till that happens is you
> will loose a lot of customers trying to make things stable again.
> I would think Denis also would have a much better informed path to take
> and I'll bet he would be happy to contract to help you. Probably money well
> spent.
>
>
>
>
> *Gino A. Villarini*
> President
> Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968
>
> On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:
>
>> We're an urban WISP with a dense mesh of wireless links and a router per
>> building. I am concerned that, without paying attention, we have grown to
>> 600+ routers and ~2550 routes in one OSPF domain. This network has a
>> diverse mix of routers from CCR1036s down to RB750UPs. We're not having any
>> OSPF problems at this time and I have plenty of other things to worry
>> about, but I'd hate to hit some limit and have the whole thing blow up.
>>
>> Does anyone have experience (positive or negative) with large flat OSPF
>> networks?
>> And, if you have had problems, what were the problems?
>>
>> Thanks,
>> Brough
>>
>> Brough Turner
>> netBlazr Inc. – Free your Broadband!
>> Mobile:  617-285-0433 <(617)%20285-0433>   Skype:  brough
>> netBlazr Inc. <http://www.netblazr.com/> | Google+
>> <https://plus.google.com/102447512447094746687/posts?hl=en> | Twitter
>> <https://twitter.com/#%21/brough> | LinkedIn
>> <http://www.linkedin.com/in/broughturner> | Facebook
>> <http://www.facebook.com/brough.turner> | Blog
>> <http://blogs.broughturner.com/> | Personal website
>> <http://broughturner.com/>
>>
>>
>>
>

Re: [AFMUG] OSPF - How large can a flat network grow?

[Josh Baird]

But you are still running MPLS on top of some IGP (OSPF, etc)?

On Thu, Jun 7, 2018 at 7:47 PM, Gino A. Villarini  wrote:

> Our network by all means is flat as all services converge into our core.
> We started by using vlans and qinq but later migrated to mpls/vpls.
>
> Now watching evpn as the next step..
>
> From: Af  on behalf of Lewis Bergman <
> lewis.berg...@gmail.com>
> Reply-To: "af@afmug.com" 
> Date: Thursday, June 7, 2018 at 3:09 PM
> To: "af@afmug.com" 
> Subject: Re: [AFMUG] OSPF - How large can a flat network grow?
>
> Gino would probably be your best source for advice as I'll bet he has one
> of the largest networks of similar construction. I think he went MPLS to
> deal with a variety of issues he was having, including OSPF. But maybe Gino
> can speak up on the issue.
> Typically, if you didn't want to do anything else, you would consider iBGP
> and break the OSPF domains up in some logical way. The big determining
> factor for me for such things is the occurrence route flapping and how
> often it happens. Route flapping will be the big indicator that you are to
> big and OSPF can't keep up. You can do some tweaking but at some point it
> just all falls apart. The bad part about waiting till that happens is you
> will loose a lot of customers trying to make things stable again.
> I would think Denis also would have a much better informed path to take
> and I'll bet he would be happy to contract to help you. Probably money well
> spent.
>
>
>
> *Gino A. Villarini*
> President
> Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968
>
> On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:
>
>> We're an urban WISP with a dense mesh of wireless links and a router per
>> building. I am concerned that, without paying attention, we have grown to
>> 600+ routers and ~2550 routes in one OSPF domain. This network has a
>> diverse mix of routers from CCR1036s down to RB750UPs. We're not having any
>> OSPF problems at this time and I have plenty of other things to worry
>> about, but I'd hate to hit some limit and have the whole thing blow up.
>>
>> Does anyone have experience (positive or negative) with large flat OSPF
>> networks?
>> And, if you have had problems, what were the problems?
>>
>> Thanks,
>> Brough
>>
>> Brough Turner
>> netBlazr Inc. – Free your Broadband!
>> Mobile:  617-285-0433 <(617)%20285-0433>   Skype:  brough
>> netBlazr Inc. <http://www.netblazr.com/> | Google+
>> <https://plus.google.com/102447512447094746687/posts?hl=en> | Twitter
>> <https://twitter.com/#%21/brough> | LinkedIn
>> <http://www.linkedin.com/in/broughturner> | Facebook
>> <http://www.facebook.com/brough.turner> | Blog
>> <http://blogs.broughturner.com/> | Personal website
>> <http://broughturner.com/>
>>
>>
>>
>

Re: [AFMUG] OSPF - How large can a flat network grow?

[Chuck McCown]

To me, flat network means all MACs are in the same broadcast domain.  
That is what I did with my first Canopy system.  
Did not even know what a VLAN was back then.  

From: Gino A. Villarini 
Sent: Thursday, June 07, 2018 5:47 PM
To: af@afmug.com 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

Our network by all means is flat as all services converge into our core.  We 
started by using vlans and qinq but later migrated to mpls/vpls.

Now watching evpn as the next step..

From: Af  on behalf of Lewis Bergman 

Reply-To: "af@afmug.com" 
Date: Thursday, June 7, 2018 at 3:09 PM
To: "af@afmug.com" 
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?


Gino would probably be your best source for advice as I'll bet he has one of 
the largest networks of similar construction. I think he went MPLS to deal with 
a variety of issues he was having, including OSPF. But maybe Gino can speak up 
on the issue.  
Typically, if you didn't want to do anything else, you would consider iBGP and 
break the OSPF domains up in some logical way. The big determining factor for 
me for such things is the occurrence route flapping and how often it happens. 
Route flapping will be the big indicator that you are to big and OSPF can't 
keep up. You can do some tweaking but at some point it just all falls apart. 
The bad part about waiting till that happens is you will loose a lot of 
customers trying to make things stable again.
I would think Denis also would have a much better informed path to take and 
I'll bet he would be happy to contract to help you. Probably money well spent.



  Gino A. Villarini
 
  President 
  Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968 




On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:

  We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

  Does anyone have experience (positive or negative) with large flat OSPF 
networks?
  And, if you have had problems, what were the problems?

  Thanks,
  Brough

  Brough Turner
  netBlazr Inc. – Free your Broadband!
  Mobile:  617-285-0433   Skype:  brough
  netBlazr Inc. | Google+ | Twitter | LinkedIn | Facebook | Blog | Personal 
website 

Re: [AFMUG] OSPF - How large can a flat network grow?

[Gino A. Villarini]

Our network by all means is flat as all services converge into our core.  We 
started by using vlans and qinq but later migrated to mpls/vpls.

Now watching evpn as the next step..

From: Af mailto:af-boun...@afmug.com>> on behalf of Lewis 
Bergman mailto:lewis.berg...@gmail.com>>
Reply-To: "af@afmug.com<mailto:af@afmug.com>" 
mailto:af@afmug.com>>
Date: Thursday, June 7, 2018 at 3:09 PM
To: "af@afmug.com<mailto:af@afmug.com>" mailto:af@afmug.com>>
Subject: Re: [AFMUG] OSPF - How large can a flat network grow?

Gino would probably be your best source for advice as I'll bet he has one of 
the largest networks of similar construction. I think he went MPLS to deal with 
a variety of issues he was having, including OSPF. But maybe Gino can speak up 
on the issue.
Typically, if you didn't want to do anything else, you would consider iBGP and 
break the OSPF domains up in some logical way. The big determining factor for 
me for such things is the occurrence route flapping and how often it happens. 
Route flapping will be the big indicator that you are to big and OSPF can't 
keep up. You can do some tweaking but at some point it just all falls apart. 
The bad part about waiting till that happens is you will loose a lot of 
customers trying to make things stable again.
I would think Denis also would have a much better informed path to take and 
I'll bet he would be happy to contract to help you. Probably money well spent.




Gino A. Villarini


President
Metro Office Park #18 Suite 304 Guaynabo, Puerto Rico 00968

[cid:aeronet-logo_310cfc3e-6691-4f69-bd49-b37b834b9238.png]

On Thu, Jun 7, 2018 at 1:40 PM Brough Turner 
mailto:bro...@netblazr.com>> wrote:
We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat OSPF 
networks?
And, if you have had problems, what were the problems?

Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc.<http://www.netblazr.com/> | 
Google+<https://plus.google.com/102447512447094746687/posts?hl=en> | 
Twitter<https://twitter.com/#%21/brough> | 
LinkedIn<http://www.linkedin.com/in/broughturner> | 
Facebook<http://www.facebook.com/brough.turner> | 
Blog<http://blogs.broughturner.com/> | Personal 
website<http://broughturner.com/>

Re: [AFMUG] OSPF - How large can a flat network grow?

[Eric Kuhnke]

Also worth mentioning that a lot of OSPF documentation available on the
Internet, makes assumptions that were valid in 2002 or so...  When a
typical router had a lot less DRAM and CPU. Such as a Cisco 3725/3745 or
even something smaller like a 2621.

Probably still true if you're trying to do OSPF on very small Mikrotiks but
not as much of a concern in the modern era. The main bottleneck in routing
platforms is FIB size and RAM for BGP tables, not so much OSPF.



On Thu, Jun 7, 2018 at 1:15 PM, Dennis Burgess 
wrote:

> This is a “unanswerable” question.  In honesty, you can have 20k routes in
> OSPF and it be responsive so the routing platform does not have a limit.
> The number of routers, is another issue, if you have 500 routers all fiber
> connected and they don’t go up/down much, then no big deal, however, if
> they do go up down quite a bit or you don’t know how to manage them, then
> yes this can be a factor. The last thing is convergence time, if you have
> lots of fiber, and/or well connected routers, then that is not a major
> issue, break part of your network and see how long it takes for a reroute,
> if that is acceptable, then again no worries.
>
>
>
> Now OSPF books, state that you should have no more than 75-100 routers,
> but I have read things that state no more than 50 and I have other networks
> that have more than 500. So..  Again, it’s not a good answerable question.
>
>
>
> However, my suggestion is to look at your network as a whole and see if
> there is some kind of logic, to splitting up your OSPF domains.  You can
> use OSPF areas, or you can use BGP between them.  But there needs to be a
> good, constant method to splitting your network like that.  Keep in mind
> that using defaults will cause traffic to shift, etc, so you need to plan
> plan plan…
>
>
>
> Just my two cents.
>
>
>
>
>
>
>
> *Dennis Burgess, Mikrotik Certified Trainer *
>
> Author of "Learn RouterOS- Second Edition”
>
> *Link Technologies, Inc* -- Mikrotik & WISP Support Services
>
> *Office*: 314-735-0270  Website: http://www.linktechs.net
>
> Create Wireless Coverage’s with www.towercoverage.com
>
>
>
> *From:* Af  *On Behalf Of *Brough Turner
> *Sent:* Thursday, June 7, 2018 1:41 PM
> *To:* af@afmug.com
> *Subject:* [AFMUG] OSPF - How large can a flat network grow?
>
>
>
> We're an urban WISP with a dense mesh of wireless links and a router per
> building. I am concerned that, without paying attention, we have grown to
> 600+ routers and ~2550 routes in one OSPF domain. This network has a
> diverse mix of routers from CCR1036s down to RB750UPs. We're not having any
> OSPF problems at this time and I have plenty of other things to worry
> about, but I'd hate to hit some limit and have the whole thing blow up.
>
> Does anyone have experience (positive or negative) with large flat OSPF
> networks?
> And, if you have had problems, what were the problems?
>
> Thanks,
> Brough
>
> Brough Turner
> netBlazr Inc. – Free your Broadband!
> Mobile:  617-285-0433   Skype:  brough
> netBlazr Inc.  | Google+
>  | Twitter
>  | LinkedIn
>  | Facebook
>  | Blog
>  | Personal website
> 
>
>
>

Re: [AFMUG] OSPF - How large can a flat network grow?

[Dennis Burgess]

This is a “unanswerable” question.  In honesty, you can have 20k routes in OSPF 
and it be responsive so the routing platform does not have a limit.  The number 
of routers, is another issue, if you have 500 routers all fiber connected and 
they don’t go up/down much, then no big deal, however, if they do go up down 
quite a bit or you don’t know how to manage them, then yes this can be a 
factor. The last thing is convergence time, if you have lots of fiber, and/or 
well connected routers, then that is not a major issue, break part of your 
network and see how long it takes for a reroute, if that is acceptable, then 
again no worries.

Now OSPF books, state that you should have no more than 75-100 routers, but I 
have read things that state no more than 50 and I have other networks that have 
more than 500. So..  Again, it’s not a good answerable question.

However, my suggestion is to look at your network as a whole and see if there 
is some kind of logic, to splitting up your OSPF domains.  You can use OSPF 
areas, or you can use BGP between them.  But there needs to be a good, constant 
method to splitting your network like that.  Keep in mind that using defaults 
will cause traffic to shift, etc, so you need to plan plan plan…

Just my two cents.



Dennis Burgess, Mikrotik Certified Trainer
Author of "Learn RouterOS- Second Edition”
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270  Website: 
http://www.linktechs.net
Create Wireless Coverage’s with www.towercoverage.com

From: Af  On Behalf Of Brough Turner
Sent: Thursday, June 7, 2018 1:41 PM
To: af@afmug.com
Subject: [AFMUG] OSPF - How large can a flat network grow?

We're an urban WISP with a dense mesh of wireless links and a router per 
building. I am concerned that, without paying attention, we have grown to 600+ 
routers and ~2550 routes in one OSPF domain. This network has a diverse mix of 
routers from CCR1036s down to RB750UPs. We're not having any OSPF problems at 
this time and I have plenty of other things to worry about, but I'd hate to hit 
some limit and have the whole thing blow up.

Does anyone have experience (positive or negative) with large flat OSPF 
networks?
And, if you have had problems, what were the problems?

Thanks,
Brough

Brough Turner
netBlazr Inc. – Free your Broadband!
Mobile:  617-285-0433   Skype:  brough
netBlazr Inc. | 
Google+ | 
Twitter | 
LinkedIn | 
Facebook | 
Blog | Personal 
website

Re: [AFMUG] OSPF - How large can a flat network grow?

[Mark Radabaugh]

As long as it’s stable you are nowhere near the limits of OSPF as a IGP, and 
there are a number of fairly simple ways to start to segment the network when 
you start to reach the limits.   OSPF itself is fine but during network 
instability situations you may run into issues with CPU on some of the 
underpowered routers which leads to further instability.   Overlays of MPLS, 
VLAN’s, or just OSPF areas can address the issues if you do want to start 
cutting the size down.

Mark

> On Jun 7, 2018, at 3:09 PM, Lewis Bergman  wrote:
> 
> Gino would probably be your best source for advice as I'll bet he has one of 
> the largest networks of similar construction. I think he went MPLS to deal 
> with a variety of issues he was having, including OSPF. But maybe Gino can 
> speak up on the issue. 
> Typically, if you didn't want to do anything else, you would consider iBGP 
> and break the OSPF domains up in some logical way. The big determining factor 
> for me for such things is the occurrence route flapping and how often it 
> happens. Route flapping will be the big indicator that you are to big and 
> OSPF can't keep up. You can do some tweaking but at some point it just all 
> falls apart. The bad part about waiting till that happens is you will loose a 
> lot of customers trying to make things stable again.
> I would think Denis also would have a much better informed path to take and 
> I'll bet he would be happy to contract to help you. Probably money well spent.
> 
> On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  > wrote:
> We're an urban WISP with a dense mesh of wireless links and a router per 
> building. I am concerned that, without paying attention, we have grown to 
> 600+ routers and ~2550 routes in one OSPF domain. This network has a diverse 
> mix of routers from CCR1036s down to RB750UPs. We're not having any OSPF 
> problems at this time and I have plenty of other things to worry about, but 
> I'd hate to hit some limit and have the whole thing blow up.
> 
> Does anyone have experience (positive or negative) with large flat OSPF 
> networks?
> And, if you have had problems, what were the problems? 
> Thanks,
> Brough
> 
> Brough Turner
> netBlazr Inc. – Free your Broadband!
> Mobile:  617-285-0433Skype:  brough
> netBlazr Inc.  | Google+ 
>  | Twitter 
>  | LinkedIn 
>  | Facebook 
>  | Blog 
>  | Personal website 
>  
> 
>  

Re: [AFMUG] OSPF - How large can a flat network grow?

[Lewis Bergman]

Gino would probably be your best source for advice as I'll bet he has one
of the largest networks of similar construction. I think he went MPLS to
deal with a variety of issues he was having, including OSPF. But maybe Gino
can speak up on the issue.
Typically, if you didn't want to do anything else, you would consider iBGP
and break the OSPF domains up in some logical way. The big determining
factor for me for such things is the occurrence route flapping and how
often it happens. Route flapping will be the big indicator that you are to
big and OSPF can't keep up. You can do some tweaking but at some point it
just all falls apart. The bad part about waiting till that happens is you
will loose a lot of customers trying to make things stable again.
I would think Denis also would have a much better informed path to take and
I'll bet he would be happy to contract to help you. Probably money well
spent.

On Thu, Jun 7, 2018 at 1:40 PM Brough Turner  wrote:

> We're an urban WISP with a dense mesh of wireless links and a router per
> building. I am concerned that, without paying attention, we have grown to
> 600+ routers and ~2550 routes in one OSPF domain. This network has a
> diverse mix of routers from CCR1036s down to RB750UPs. We're not having any
> OSPF problems at this time and I have plenty of other things to worry
> about, but I'd hate to hit some limit and have the whole thing blow up.
>
> Does anyone have experience (positive or negative) with large flat OSPF
> networks?
> And, if you have had problems, what were the problems?
>
> Thanks,
> Brough
>
> Brough Turner
> netBlazr Inc. – Free your Broadband!
> Mobile:  617-285-0433 <(617)%20285-0433>   Skype:  brough
> netBlazr Inc.  | Google+
>  | Twitter
>  | LinkedIn
>  | Facebook
>  | Blog
>  | Personal website
> 
>
>
>